Setting up Applications

IIS 6 Optimizations: Connection Timeout, Application Pools, Threads, Connection Strings, and File Compression.

Alex Kuo — Wed, 28 Oct 2009 21:00:57 GMT

One of the sites I maintain was unable to serve pages due to the server timing out. At first, I suspected it was a bandwidth issue, since traffic was significantly elevated due to a recent ad campaign that was initiated. On close examination, I noticed the system's resources were barely taxed at all. Hard disk I/0, CPU usage, bandwidth, and memory were either normal or barely used.

I then remembered a similar incident with another site that showed the same symptoms, except the site was hosted on an Apache server. The problem was that Apache was consuming too much memory and was unable to fulfill requests fast enough due to having a high connection timeout setting. The fix was to limit the amount of processes spawned and reduce the connection timeout setting.

Since memory and CPU usage was not an issue, the only other suspect was to check the connection timeout setting. The Connection Timeout setting is located under:

Open IIS Manager
Expand websites
Select target website's properties.
Select Website tab

On default, it is set to 120 or 900 seconds – a very long time, especially if a web server has a queue of 1000+ objects to serve. My recommendation is to set the Connection Timeout setting to 2-5 seconds. This will free up the amount of connections available to users immensely. In addition, there are other optimizations to threading and compression that aren't enabled by default.

The following tweaks will help manage processes and threads for serving pages. In the %WINDIR%\Microsoft.NET\Framework\v1.1.4322\CONFIG\machine.config, set the following:

Setting	Default	"Optimized"
maxconnection	2	12 * #CPUs
maxIoThreads	20	100
maxWorkerThreads	20	100
minFreeThreads	8	88 * #CPUs
minLocalRequestFreeThreads	4	76 * #CPUs

Enabling file compression can free up a significant amount of bandwidth and increase the number of objects served at a time. First, give the folder where the compressed files will be stored write permissions to the IIS User. By default, this is “%windir%\IIS Temporary Compressed Files”. The IUSR_{machinename} will need the write permission to the folder.

Next, you need to enable Compression in IIS.

Open IIS Manager
Select the websites folder
Right click and select properties.
Select the Service's tab
Enable Compress application files
Enable Compress static files
Set max size to something sensible.

Then enable the direct Metabase editing. This allows you to edit the metabase while IIS is running.

Open IIS Manager
Right click the computer icon
select properties
Check the checkbox 'Enable Direct Metabase Edit'

Afterwards, you will need to open the IIS metabase by editing the file, %WINDIR%\system32\inetsrv\metabase.xml. Then find the attribute HcDynamicCompressionLevel and change the setting from 0 to 9, zero is the default setting. This will compress the pages as much possible without straining the CPU too much. There will two HcDynamicCompressionLevel attributes, each one under IIsCompressionScheme element that set deflate and gzip behaviors. Save the file.

Next entails adding the file extensions for static and dynamic pages that will be compressed. To do this, open the command prompt and open the directory C:\InetPub\AdminScripts\. Then execute the following to enable compression for the most commonly served files.

cscript.exe adsutil.vbs set W3Svc/Filters/Compression/GZIP/HcFileExtensions "htm" "html" "txt" "ppt" "xls" "xml" "pdf" "xslt" "doc" "xsl" "htc" "js" "css"

cscript.exe adsutil.vbs set W3Svc/Filters/Compression/DEFLATE/HcFileExtensions "htm" "html" "txt" "ppt" "xls" "xml" "pdf" "xslt" "doc" "xsl" "htc" "js" "css"

cscript.exe adsutil.vbs set W3Svc/Filters/Compression/GZIP/HcScriptFileExtensions "asp" "dll" "exe" "aspx" "asmx" "ashx"

cscript.exe adsutil.vbs set W3Svc/Filters/Compression/DEFLATE/HcScriptFileExtensions "asp" "dll" "exe" "aspx" "asmx"

IISreset.exe /restart

Do not compress images, as they are already compressed [jpg, gif, tiff, etc.]

Adjusting the IIS Application Pool can help resource utilization. The settings I changed were:

'Limit Request Queues' setting under the Performance tab. Depending on how many CPUs, Worker Processes, and memory available, this setting can increase performance. The machine I was tweaking this setting on had 4 CPUs and 8 GB of RAM. I increased the limit from 4000 to 10,000 with only 1 worker process enabled.
Another setting I disabled in the tab was the 'Shutdown worker processes after being idle for..'. I noticed that starting worker processes was extremely expensive in terms of CPU usage.

If your site is database driven, you may want to turn off Connection Pooling and set a low connection timeout if you're confident that your server will not have resources issues when a large number of database connections are opened. In ASP.NET, add the Connect Timeout and Pooling settings to the connection string:

Data Source=SERVERNAME;Database=DATABASENAME;Integrated Security=SSPI;Pooling=false;Connect Timeout=5;

For references, please consult the following Links

Ubuntu Server 7.10: Installing Proftpd and 'warning: unable to determine IP address of'

Thu, 06 Mar 2008 23:54:24 GMT

I had some trouble installing proftpd today on a Linode.com VPS server. For anyone that hasn't used linode, it's a colo service that lets people lease different grades of Linux VPS servers at reasonable prices. I've been using them for 4 months with few problems.

Installing proftpd in Ubuntu is easy. All you have to do is open a Terminal window or SSH into your server and type:

sudo apt-get install proftpd

However for some reason, the Ubuntu distros mounted on Linode's servers have a quirk or two that require a few tweaks to get the server working. The first thing is whether to run the server in standalone or xinetd mode. The server won't be able access port 21 unless it's operating in standalone mode. So, set it to standalone mode.

The second quirk, which is not unique to Linode VPS servers, regards resolving the host's address. You must add the server's static address to the hosts file. For example, if your machine's name is 'johnny' and the static IP address is 79.221.23.12.

To edit your hosts file type

sudo vi /etc/hosts

Your hosts file should look like this:

127.0.0.1       localhost
79.221.23.12	johnny

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

You can determine your machine's host name by opening the file, '/etc/hostname'.

Aftewards, check your proftpd configuration. To do this, type:

sudo proftpd -td5

If you would like more information on setting up Proftpd or Ubuntu's network configuration. Please consult the following:

PostgreSQL 8.2: Recovering from a Corrupted Database

Tue, 27 Nov 2007 01:50:31 GMT

Today I encountered my second database corruption or crash on my development machine. The problem is that my IDE drive sucks, an old IBM 75GXP. For those of you who don't remember, these drives were notorious for having all sorts of problems when they were released. Most of the problems revolved around dying after six months. Regardless, I'm still using this thing for another month.

You can tell if your database became corrupted by restarting the postmaster or typing:

sudo /etc/init.d/posrgresql-8.2 restart

The result of the command will look similar to the following:

2007-11-26 13:14:48 CST LOG:  server process (PID 16312) was terminated by signal 11
2007-11-26 13:14:48 CST LOG:  terminating any other active server processes
2007-11-26 13:14:48 CST LOG:  all server processes terminated; reinitializing
2007-11-26 13:14:48 CST LOG:  database system was interrupted at 2007-11-26 13:14:47 CST
2007-11-26 13:14:48 CST LOG:  checkpoint record is at 0/965E4130
2007-11-26 13:14:48 CST LOG:  redo record is at 0/965E4130; undo record is at 0/0; shutdown TRUE
2007-11-26 13:14:48 CST LOG:  next transaction ID: 0/73629; next OID: 10952256
2007-11-26 13:14:48 CST LOG:  next MultiXactId: 1; next MultiXactOffset: 0
2007-11-26 13:14:48 CST LOG:  database system was not properly shut down; automatic recovery in progress
2007-11-26 13:14:48 CST LOG:  record with zero length at 0/965E4178

Before initiating any recovery options, I recommend backing up your $PGDATA directory. The following directions will attempt to reset the transaction log, and you may loose the latest changes to your databases. The first option for recovering from a crash is to open your '/etc/postgresql/8.2/main/postgresql.conf' file and turn on the following option:

...
zero_damaged_pages = true

After saving the changes, type the following to restart postgresql:

sudo /etc/init.d/postgresql-8.2 restart

If you still see error messages, you may need to restart the entire computer in order to get postgresql to start.

After restarting postgresql, reopen the '/etc/postgresql/8.2/main/postgresql.conf' file and comment out the zero_damaged_pages option. The line should look similar to the following:

#zero_damaged_pages = true

After reading a few posts, it seems that IDE drives in general should not be used when working with databases because of corruption issues. If you are using an IDE drive, I recommend adding the following options to the '/etc/postgresql/8.2/main/postgresql.conf' file.

fsync = on				# turns forced synchronization on or off
wal_sync_method = fsync

The problem with turning on fsync is that it slows down the performance of Postgresql, however, it should prevent further corruption issues with your database.

You can find more information about these configuration options in the Postgresql 8.2 documentation.

Reference:

pg_resetxlog - Tool used as a last resort for resetting the transaction log

Ubuntu Desktop 7.10: Setup HP 1200 Printer

Wed, 24 Oct 2007 10:09:03 GMT

After a short search through the Ubuntu forums, I ran into this post that went into details about setting up a HP printing device. After briefly reading through the instructions, I ran a utility called 'HPLIP'. 'HPLIP' is a program that will automatically download and compile all the necessary files to activate your printer. The program will ask you a few questions about your computer and request you replug-in your printer at the end of the installation. After doing this, I printed a test page and golly... it actually worked.

Ubuntu Linux: Syncing Documents between Different Computers Using NFS and Unison

Thu, 18 Oct 2007 07:22:50 GMT

The other day I successfully made a full transition from my laptop to my desktop as my primary development environment. The biggest hurdle before completing this transition was transferring and syncing documents between my two laptop and desktop. For quick file transfers, I created a network file share, or NFS, on my desktop, while mounting the drive on my laptop. For a quick overview on how to setup and mount NFS, consult this thread on Ubuntu forums.

I also wanted to sync and compare documents from a centralized server and have the ability to compare differences between a client and a centralized 'master' copy. (think Subversion - but without all the permissions and change logging) After a quick Google search, I found a wonderful program called 'Unison'. This program will allow a user to define a master directory on a server and slave directory on a client. Master being the label for the directory where all clients compare their files and slave as clients that send new files or receive files copied from other clients to the master directory. For directions on installing unison, consult this article on howtoforge.com.

Ubuntu Desktop: Unplugging/Replugging your Network Cable on your Laptop and Requesting a New Address from DHCP.

Fri, 10 Aug 2007 03:05:53 GMT

Sometimes when working on my laptop, which has Ubuntu Desktop installed on it, I have to move it around and therefore unplug the network cable and switch to wireless or vice versa. Unplugging and replugging your laptop into a network sometimes results in the laptop's inability to renew its IP address or re-establish a connection with the internet. After a quick Google search, I found this post about the problem.

In order to issue a command similar to 'ipconfig -renew' in Windows, open your shell and type the following:

sudo ifdown eth0
sudo ifup eth0

These two commands will renew your IP address and should fix the connection problem. However, there's a program called 'ifplugd' that monitors your network connection and automatically renews your address if this problem occurs. To install this program, open your shell and type:

sudo apt-get install ifplugd

Mirroring an FTP site in Ubuntu Server

Mon, 09 Jul 2007 12:11:27 GMT

The other day I was tasked with mirroring a FTP site, about 5+ gigs of files, on our local server. Mirroring directories is a fairly common task when administering servers, however, the main differences when tasked with this job are the protocols available, whether the job is bi-directional or one way, and the how fast the mirroring needs to occur.

Lucky for me, this job did not demand instantaneous sync and the job was only one way - meaning changes from a server were reflected only from one server. The biggest problem was this job was limited to using only the FTP protocol for mirroring the site. This immediately removed rsync, a popular server/client for syncing directories remotely, as an option. After a quick search through the Ubuntu forums, I stumbled upon a post that detailed several programs on mirroring an FTP site using FTP protocol only. I chose a program called 'ftpmirror'.

Ftpmirror is a program that lets a user define 'packages', which are configuration details for mirroring an FTP site, and scheduling these 'packages' to be run daily, monthly, or weekly. To install this program, I typed

sudo apt-get install ftpmirror

If you're using Ubuntu Server, the configuration files should reside in the '/etc/ftpmirror/' directory. Upon browsing through the directory, you will find a file called 'ftpmirror.cf-sample'. This file contains a few example 'packages' that can be used as a template. The user puts any active 'packages' in the 'ftpmirror.cf' file. My 'ftpmirror.cf' file looks like this:

package = alexkuo_media
 ftp-server = master.alexkuo.info
 ftp-user = mirror
 ftp-pass = password
 remote-directory = /media/pics/
 local-directory = /home/deploy/media/pics

This package uses the directory '/media/pics/' as the root directory on the ftp server 'master.alexkuo.info' and uses the login/password, mirror/password, to login into the remote server. All files, directories, and subdirectories found in the '/media/pics' directory are then downloaded into the '/home/deploy/media/pics' directory on the local machine once the package is activated.

I decided to run this job once a week, so I added the package to the '/etc/ftpmirror/list.weekly' file. To do this, open the 'list.weekly' file with a text editor. Mine looks like this:

alexkuo_media

Pretty plain huh? I removed the comments that originally came with the file, so it looks pretty bare. Adding another package involves defining another package in the ftpmirror.cf file and appending the package name on a new line in one of the *.list files.

Ubuntu Server 7.04 Fiesty: Mounting or Accessing a USB Drive in Bash Shell

Fri, 29 Jun 2007 02:32:15 GMT

I ran into a problem today accessing a USB Drive in Ubuntu Server. In Windows and in Ubuntu Standard, plug and play works fine. My laptop, which has Ubuntu installed, immediately recognized the USB drive once it was plugged in. I accessed its contents in the Bash shell by typing:

ls -l /media
ls -l /media/ExternalHDD

Typing the same thing in Ubuntu Server will not yield the same results. Ubuntu 7.04 server does not come with automatic mounting. After plugging in your drive, you need to run a few commands before access to the drive will be allowed. The first command is to install a program called 'pmount'. Type:

sudo apt-get install pmount

After installing the pmount, you need to figure out what the drive is called on your system. Type:

fdisk -l

You should see output that looks like this:

Disk /dev/sdf: 300.0 GB, 300069052416 bytes
255 heads, 63 sectors/track, 36481 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

   Device Boot      Start         End      Blocks   Id  System
/dev/sdf4   *           1       36481   293033601    7  HPFS/NTFS

The USB drive I am using is called '/dev/sdf4'. To mount the drive, you're need to create a directory to alias the drive and use pmount to reference the alias. To do this I type:

sudo mkdir /media/usbdrive
pmount /dev/sdf4 /media/usbdrive

Afterwards, I copied a bunch of files on the USB drive and set the ownership of the files from root to myself. I typed:

sudo cp -R /media/usbdrive/filesToCopy /home/alex/
cd /home/alex
sudo chown -R alex:alex filesToCopy

New Article

CruiseControl Server and Django: How to Setup CruiseControl to Automatically Deploy Django Applications

Ubuntu Server 7.04 Fiesty: Installation Problems and Changing Network Settings from DHCP to Static IP

Tue, 26 Jun 2007 13:12:31 GMT

Tonight I deployed another installation of Ubuntu Server 7.04 Fiesty. The biggest problem I ran into was setting up the network card. Apparently, the Ubuntu setup wizard tries to contact the network in at least two points during the installation wizard. In one of the points of the installation, failure to contact a server on the internet results in the installation hanging. I got around this by running the installation while connected to my residential DHCP network and then redeploying the server to the DMZ, which requires defining a static IP.

This created another problem - How do you change the network settings from DHCP to static IP address? In order to solve this problem, you need to update your interfaces file to use an assigned static IP address and configure the server to use your network's DNS servers.

To change your network settings from DHCP to static IP, type:

sudo vi /etc/network/interfaces

The file displayed should have a line that says ' iface eth0 inet dhcp' or something similar. Change the file to look like the following:

....
....
....
auto eth0
iface eth0 inet static
	address 71.164.212.40
	netmask 255.255.255.0
	network 71.164.212.0
	broadcast 71.164.0.255
	gateway 71.164.212.1

Afterwards, restart your network interface by typing:

sudo /etc/init.d/networking restart

Next, you need to configure your DNS servers. You need at least two entries. To edit your DNS entries type:

sudo vi /etc/resolv.conf

My file looked like this:

nameserver 4.2.2.1
nameserver 4.2.2.2

After saving the file, you should be able to connect to the network.

If you would like to edit your hosts file, type the following in your shell.

sudo vi /etc/hosts

You can add more references to your localhost in here. Mine looks like this:

127.0.0.1       localhost
127.0.0.1	alex-server
127.0.0.1	sandbox-server

# The following lines are desirable for IPv6 capable hosts
::1     ip6-localhost ip6-loopback
fe00::0 ip6-localnet
ff00::0 ip6-mcastprefix
ff02::1 ip6-allnodes
ff02::2 ip6-allrouters
ff02::3 ip6-allhosts

If you would like to change the hostname of your server, type the following:

sudo vi /etc/hostname

After typing in your new hostname, you need to run a script to implement the change. Type:

sudo /etc/init.d/hostname.sh start

References

Ubuntugeek.com - Ubuntu 7.04 Fiesty Fawn LAMP Server Setup

Getting Around Linux: Compiling Python Extensions and Ubuntu's Missing Packages

Fri, 18 May 2007 21:40:12 GMT

Last night I spent an awful lot of time trying to figure out how to compile an extension for Python. My struggles were quickly solved with the help of a charitable poster on Ubuntuforums.org. On default installations of Ubuntu 7.04 Fiesty, no development packages for the C compiler (aka gcc) and Python are included. In the end, the remedy was to install the C compiler and the Python development packages.

In order to accomplish this, type the following in the Bash prompt.

sudo apt-get install python-dev

This installs the Python development packages. You also need to type:

sudo apt-get install build-essential

This installs the C compiler.

After adding these magical packages, the extension I was trying to add compiled and was 'successfully' working. I put 'successfully' in quotes because I later found out that their were errors in the extension regarding a function call to localization, which was fixed about an hour later after some debugging.

Just in case you were wondering, the extension I was adding is called pgasync. It's a asynchronous data adapter for PostgreSQL. If this works reliably, it may be able to scale better than using psycopg2, which is synchronous.