Nginx vs OpenLiteSpeed vs Cherokee Benchmarks

Below are very quick Siege benchmark comparisons against the basic phpinfo php page for three different web servers and their respective PHP handlers. Centmin Mod's Nginx server, OpenLiteSpeed server (open source version of LiteSpeed Enterprise) and Cherokee web server.

Note it's first time using Cherokee and it's the reason for this benchmark comparison to investigate the claim made by a Centmin Mod user Anuj on our Centmin Mod Google+ Community who mentioned that Cherokee was faster than both LiteSpeed and Nginx in his initial tests. But I am no stranger to using Nginx with Centmin Mod or OpenLiteSpeed (blog articles).

So I read the Cherokee documentation and it seems by default out of the box installs, PHP is configured with Cherokee front line caching enabled for dynamic PHP requests. Note exactly valid comparison between other web server's and their default non-cached PHP requests. So I ran some benchmarks below to see how Cherokee faired with PHP cache on and off in comparison to Nginx and OpenLiteSpeed.

Server Configuration
PHP info and locations
Source Configure and Compile Options
Siege benchmark v3.0.2 Results
- Table Comparison
- Raw results below
Conclusion

Server Configuration

VirtualBox CentOS 6.4 64bit
Xeon W3540 @3.2Ghz allocated 4 cpu threads
1500MB memory
20GB disk on 1TB Western Digital Black Caviar

PHP info and locations

Centmin Mod Nginx 1.4.2 phpinfo.php (php-fpm 5.4.18 + APC Cache 3.1.13) = http://192.168.0.199/phpinfo.php
OpenLiteSpeed 1.2.4 phpinfo.php (php-fpm 5.4.18 + APC Cache 3.1.13) = http://192.168.0.199:8088/phpinfo.php
Cherokee 1.2.103 phpinfo.php (php-fpm 5.4.18 + APC Cache 3.1.13) = http://192.168.0.199:81/phpinfo.php

Source Configure and Compile Options

All 3 web servers, are configure/compiled via source install.

For Centmin Mod based Nginx

  nginx -V
nginx version: nginx/1.4.2
built by gcc 4.4.7 20120313 (Red Hat 4.4.7-3) (GCC) 
TLS SNI support enabled
configure arguments: --sbin-path=/usr/local/sbin --conf-path=/usr/local/nginx/conf/nginx.conf --add-module=../ngx_pagespeed-release-1.6.29.5-beta --with-http_spdy_module --with-http_ssl_module --with-http_gzip_static_module --with-http_stub_status_module --with-http_sub_module --with-http_addition_module --with-http_secure_link_module --with-http_flv_module --with-http_realip_module --with-libatomic --with-pcre=../pcre-8.33 --with-pcre-jit --with-http_dav_module --add-module=../nginx-dav-ext-module-0.0.2 --add-module=../ngx-fancyindex-ngx-fancyindex --add-module=../ngx_cache_purge-2.1 --add-module=../nginx-accesskey-2.0.3 --with-openssl=../openssl-1.0.1e --add-module=../nginx-http-concat-master --with-openssl-opt=enable-tlsext

For OpenLiteSpeed default ./configure option

  ./configure
make
make install

For Cherokee

  cherokee -i
Compilation
 Version: 1.2.103
 Compiled on: Aug 24 2013 12:21:34
 Arguments to configure:  '--localstatedir=/var' '--prefix=/usr' '--sysconfdir=/etc' '--with-wwwroot=/var/www' 'CC=ccache gcc'

Installation
 Deps dir: /usr/share/cherokee/deps
 Data dir: /usr/share/cherokee
 Icons dir: /usr/share/cherokee/icons
 Themes dir: /usr/share/cherokee/themes
 Plug-in dir: /usr/lib/cherokee
 Temporal dir: /tmp

Plug-ins
 Built-in: 

Support
 IPv6: yes
 Pthreads: yes
 Tracing: no
 sendfile(): yes
 syslog(): yes
 Polling methods: epoll poll select 
 SSL/TLS: libssl
 TLS SNI: yes

http header comparisons

curl -I http://192.168.0.199/phpinfo.php
curl -I http://192.168.0.199:8088/phpinfo.php
curl -I http://192.168.0.199:81/phpinfo.php

  curl -I http://192.168.0.199/phpinfo.php
Server: nginx centminmod
Date: Sat, 24 Aug 2013 03:33:05 GMT
Content-Type: text/html; charset=utf-8
Connection: keep-alive
Vary: Accept-Encoding

  curl -I http://192.168.0.199:8088/phpinfo.php
HTTP/1.1 200 OK
Content-type: text/html
Date: Sat, 24 Aug 2013 03:33:05 GMT
Accept-Ranges: bytes
Server: LiteSpeed

  curl -I http://192.168.0.199:81/phpinfo.php
HTTP/1.1 200 OK
Connection: close
Date: Sat, 24 Aug 2013 03:33:06 GMT
Server: Cherokee
Content-type: text/html

Siege benchmark v3.0.2 Results

Run Siege benchmark v3.0.2 with following parameters:

siege -q -b -c25 -r50 http://192.168.0.199/phpinfo.php
siege -q -b -c25 -r50 http://192.168.0.199:8088/phpinfo.php
siege -q -b -c25 -r50 http://192.168.0.199:81/phpinfo.php

Notes:

1. Number of workers or threads the web server uses will effect the results. Centmin Mod Nginx by default only uses 1 worker_process. OpenLiteSpeed also defaults to blank number of workers which sets it to 1 worker. While Cherokee defaults to automatically determine number of threads based on cpu thread count which for this test server defaulted to 4 cpu threads.

Nginx with 4 workers

  ps aufx | grep nginx | grep -v grep
root      3347  0.0  0.1  51820  1596 ?        Ss   15:40   0:00 nginx: master process /usr/local/sbin/nginx -c /usr/local/nginx/conf/nginx.conf
nginx     3354  0.1  0.1  52204  2296 ?        S<   15:40   0:00  \_ nginx: worker process                                    
nginx     3355  0.1  0.1  52204  2296 ?        S<   15:40   0:00  \_ nginx: worker process                                    
nginx     3356  0.1  0.1  52204  2296 ?        S<   15:40   0:00  \_ nginx: worker process                                    
nginx     3357  0.1  0.1  52204  2280 ?        S<   15:40   0:00  \_ nginx: worker process

OpenLiteSpeed with 4 workers

  ps aufx | grep litespeed | grep -v grep     
root      3370  0.1  0.2  59312  3196 ?        S    15:40   0:00 openlitespeed (lshttpd - main)
root      3374  0.0  0.0  58156  1212 ?        S    15:40   0:00  \_ openlitespeed (lscgid)
nobody    3375  0.0  0.1  59216  1900 ?        S    15:40   0:00  \_ openlitespeed (lshttpd - #01)
nobody    3376  0.1  0.1  59216  1900 ?        S    15:40   0:00  \_ openlitespeed (lshttpd - #02)
nobody    3377  0.0  0.1  59216  1900 ?        S    15:40   0:00  \_ openlitespeed (lshttpd - #03)
nobody    3378  0.0  0.1  59216  1900 ?        S    15:40   0:00  \_ openlitespeed (lshttpd - #04)

Cherokee with specifically set threads to 4 shows the same as when default number of threads set.

  ps aufx | grep cherokee | grep -v grep
root      3474  0.0  0.2 283164  3656 pts/0    Sl+  15:41   0:00      |   \_ cherokee-admin -b 192.168.0.199 -p 9090
root      3487  0.4  1.2 425484 18816 pts/0    Sl+  15:41   0:01      |       \_ python2 /usr/share/cherokee/admin/server.py 4000 /etc/cherokee/cherokee.conf
root      3591  0.0  0.0  24364   628 ?        Ssl  15:42   0:00 /usr/sbin/cherokee -d -C /etc/cherokee/cherokee.conf
root      3887  0.0  0.2 277148  3140 ?        Sl   15:46   0:00  \_ /usr/sbin/cherokee-worker  -C /etc/cherokee/cherokee.conf

2. Notice the difference in the Longest transaction time between Centmin Mod Nginx, OpenLiteSpeed and Cherokee web servers for Siege benchmark run against phpinfo php page. Might indicate where Cherokee was serving a non-cached response ?

3. Data transferred lower for OpenLiteSpeed due to difference default compression levels ? Nginx level 6, OpenLiteSpeed 6 for dynamic content and Cherokee just has default labelled.

4. Shortest transaction time in Siege only reports to 2 decimal places.

5. Cherokee default PHP cache enabled setting.

With PHP cache disabled

6. All three have almost the same PHP configure options compiled. Cherokee uses same php-fpm server as Centmin Mod Ngixn so essentially same config. OpenLiteSpeed needs LSAPI PHP compiled and I used fairly similar configure options (outlined below). I make no claim for these results to be 100% apple to apple comparisons seeing as I am still a first time Cherokee user. But these results are basically default install settings for respective Centmin Mod Nginx, OpenLiteSpeed and Cherokee setups.

Centmin Mod Nginx and Cherokee php-fpm configure options:

  '../configure' '--enable-cgi' '--enable-fpm' '--enable-pcntl' '--with-mcrypt' '--with-mhash' '--with-zlib' '--with-gettext' '--enable-exif' '--enable-zip' '--with-bz2' '--enable-soap' '--enable-sockets' '--enable-sysvmsg' '--enable-sysvsem' '--enable-sysvshm' '--enable-shmop' '--with-pear' '--enable-mbstring' '--with-openssl' '--with-mysql=mysqlnd' '--with-libdir=lib64' '--with-mysqli=mysqlnd' '--with-mysql-sock=/var/lib/mysql/mysql.sock' '--with-curl' '--with-gd' '--with-xmlrpc' '--enable-bcmath' '--enable-calendar' '--enable-ftp' '--enable-gd-native-ttf' '--with-freetype-dir=lib64' '--with-jpeg-dir=lib64' '--with-png-dir=lib64' '--with-xpm-dir=lib64' '--enable-pdo' '--with-pdo-sqlite' '--with-pdo-mysql=mysqlnd' '--enable-inline-optimization' '--with-imap' '--with-imap-ssl' '--with-kerberos' '--with-readline' '--with-libedit' '--with-fpm-user=nginx' '--with-fpm-group=nginx' '--with-config-file-scan-dir=/root/centminmod/php.d'

OpenLiteSpeed LSAPI PHP configure options:

  './configure' '--prefix=/usr/local/lsws/lsphp5' '--with-mcrypt' '--with-mhash' '--with-zlib' '--with-gettext' '--enable-exif' '--enable-zip' '--with-bz2' '--enable-soap' '--enable-sockets' '--enable-sysvmsg' '--enable-sysvsem' '--enable-sysvshm' '--enable-shmop' '--with-pear' '--enable-mbstring' '--with-openssl' '--with-mysql=mysqlnd' '--with-libdir=lib64' '--with-mysqli=mysqlnd' '--with-mysql-sock' '--with-curl' '--with-gd' '--with-xmlrpc' '--enable-bcmath' '--enable-calendar' '--enable-ftp' '--enable-gd-native-ttf' '--with-freetype-dir=lib64' '--with-jpeg-dir=lib64' '--with-png-dir=lib64' '--with-xpm-dir=lib64' '--enable-pdo' '--with-pdo-sqlite' '--with-pdo-mysql=mysqlnd' '--enable-inline-optimization' '--with-imap' '--with-imap-ssl' '--with-kerberos' '--enable-posix' '--enable-xml' '--with-libexpat-dir=/usr' '--with-config-file-scan-dir=/openlitespeed/php.d' '--with-litespeed'

Table Comparison

Raw results below

Centmin Mod Nginx default with 1 worker_process on 4 cpu thread server

  siege -q -b -c25 -r50 http://192.168.0.199/phpinfo.php

Transactions:                   1250 hits
Availability:                 100.00 %
Elapsed time:                   2.92 secs
Data transferred:              14.65 MB
Response time:                  0.06 secs
Transaction rate:             428.08 trans/sec
Throughput:                     5.02 MB/sec
Concurrency:                   24.43
Successful transactions:        1250
Failed transactions:               0
Longest transaction:            0.11
Shortest transaction:           0.01

Centmin Mod Nginx change default from 1 to 4 worker_process on 4 cpu thread server

  siege -q -b -c25 -r50 http://192.168.0.199/phpinfo.php

Transactions:                   1250 hits
Availability:                 100.00 %
Elapsed time:                   2.05 secs
Data transferred:              14.65 MB
Response time:                  0.04 secs
Transaction rate:             609.76 trans/sec
Throughput:                     7.15 MB/sec
Concurrency:                   24.53
Successful transactions:        1250
Failed transactions:               0
Longest transaction:            0.11
Shortest transaction:           0.01

OpenLiteSpeed default with 1 worker (number of works is left blank out of box and defaults to 1)

  siege -q -b -c25 -r50 http://192.168.0.199:8088/phpinfo.php

Transactions:                   1250 hits
Availability:                 100.00 %
Elapsed time:                   2.89 secs
Data transferred:              12.67 MB
Response time:                  0.06 secs
Transaction rate:             432.53 trans/sec
Throughput:                     4.38 MB/sec
Concurrency:                   24.28
Successful transactions:        1250
Failed transactions:               0
Longest transaction:            0.10
Shortest transaction:           0.01

OpenLiteSpeed changed default number of workers from 1 to 4

  siege -q -b -c25 -r50 http://192.168.0.199:8088/phpinfo.php

Transactions:                   1250 hits
Availability:                 100.00 %
Elapsed time:                   1.41 secs
Data transferred:              12.67 MB
Response time:                  0.03 secs
Transaction rate:             886.52 trans/sec
Throughput:                     8.98 MB/sec
Concurrency:                   23.39
Successful transactions:        1250
Failed transactions:               0
Longest transaction:            0.10
Shortest transaction:           0.00

Cherokee allow PHP caching default out of box.

Cherokee has a note next to this setting:

This caching mechanism should still be considered an experimental technology. Use it with caution.

  siege -q -b -c25 -r50 http://192.168.0.199:81/phpinfo.php

Transactions:                   1250 hits
Availability:                 100.00 %
Elapsed time:                   2.29 secs
Data transferred:              14.50 MB
Response time:                  0.05 secs
Transaction rate:             545.85 trans/sec
Throughput:                     6.33 MB/sec
Concurrency:                   24.67
Successful transactions:        1250
Failed transactions:               0
Longest transaction:            0.13
Shortest transaction:           0.02

Cherokee change to forbid PHP caching option to disable caching.

Look at uncached run's Longest transaction time very high at 4.72 seconds

  siege -q -b -c25 -r50 http://192.168.0.199:81/phpinfo.php

Transactions:                   1249 hits
Availability:                  99.92 %
Elapsed time:                   5.00 secs
Data transferred:              14.53 MB
Response time:                  0.06 secs
Transaction rate:             249.80 trans/sec
Throughput:                     2.91 MB/sec
Concurrency:                   13.97
Successful transactions:        1249
Failed transactions:               1
Longest transaction:            4.72
Shortest transaction:           0.01

And even with Cherokee PHP cache disabled, it still fluctuates between runs from 249 requests/s to 573 requests/s.

  siege -q -b -c25 -r50 http://192.168.0.199:81/phpinfo.php

Transactions:                   1250 hits
Availability:                 100.00 %
Elapsed time:                   2.18 secs
Data transferred:              14.50 MB
Response time:                  0.04 secs
Transaction rate:             573.39 trans/sec
Throughput:                     6.65 MB/sec
Concurrency:                   24.52
Successful transactions:        1250
Failed transactions:               0
Longest transaction:            0.12
Shortest transaction:           0.00

Conclusions

What conclusions can be made ? From my experience using LiteSpeed Enterprise, OpenLiteSpeed, Apache and Nginx web servers, I already knew for best scalability and performance in high concurrency user loads, LiteSpeed's LSAPI PHP implement will always win when it comes to non-cached PHP request serving. I have worked with vBulletin forums pushing up to 14,000+ non-cached PHP requests per second via LiteSpeed Enterprise LSAPI PHP without any problems. I wrote about my thoughts on the matter on my blog several years ago.

Nginx also can be configured to cache it's PHP requests using fastcgi_cache. Example setup for Centmin Mod and Wordpress with fastcgi_cache. So for proper comparison with default Cherokee PHP cache enabled setup, you would need to compare with Nginx and php-fpm configured with fastcgi_cache.
Litespeed Enterprise server can do the same via LiteSpeed Cache.
Unfortuantely, OpenLiteSpeed free open source version of LiteSpeed Enterprise has it's LiteSpeed Cache support removed. But it looks like it performs fairly well with non-cached PHP requests already from above benchmarks. You would need Litespeed Enterprise if you want to use LiteSpeed Cache and cache PHP requests.
Cherokee seems to be default have PHP caching enabled although they do have a note that mentions it's experimental in nature and to use with caution. They also mention that by default front line cache doesn't cache results with cookies. So not sure if phpinfo.php PHP file is the best choice to benchmark against versus PHP files and PHP apps which utilise cookies extensively ? No idea why even with PHP cache disabled, that Siege benchmarks would fluctuate between non-cached and cached like results ? If any Cherokee users know, please let me know via Centmin Mod Google+ Community.

Again, this quick benchmark session was just to confirm Anuj's claims about Cherokee's PHP performance against Nginx and LiteSpeed as I was curious about this and set out to do my own tests. If I have more time in future I will do more tests. If you have any comments or corrections to my above assumptions, please let me know via Centmin Mod Google+ Community

Nginx vs OpenLiteSpeed vs Cherokee Benchmarks