Recently on LinkedIn in a "SharePoint – Enterprise Solutions Architects" group discussion I responded to the following question about Web App vs. Site Collection in MOSS. Several people found my comments helpful, so I decided to open it up to a wider audience.
MOSS 2007 Web Application vs. Site Collection – Pros and cons? When should one consider creating a separate web application? Search behavior on separate Web Applications?
A separate web collection is good for a separate portal with its own hostname. For example you can have two portals on the same farm http://benefits (paycheck, health benefits, and other HR forms) and http://knowledgebase (information portal about products and services) both would be separate portals with separate purposes. By doing this, you could move one of the web applications to a different farm if needed. Also, you by using separate web applications, you could assign these to different Shared Service Providers (SSP) which would allow different search contexts, BDC connections, and other shared services such as Excel Services (MOSS 2007 Enterprise). So from a licensing standpoint, you could control cost if you only needed certain users access to enterprise features. By default, each web application will have its own content database.
If you wanted to have a single portal (on a single web application), you could put the HR Benefits on one site collection and the knowledge base on a different site collection and get the same isolation from a security standpoint, and even put them in separate content databases, but since they are in the same web application they will end up sharing the same hostname of the URL and must share the same SSP (a limitation in MOSS 2007). So you might have http://portal/hr/benefits and http://portal/knowlegebase as your separate site collections. Note that this limitation of a single SSP applied to the entire web application has changed in SharePoint 2010 due to the new Service Application model.
A few other differences:
1. Each web application has its own IIS web site (thus can have separate App Pool and .NET application domain) From a scalability stand point, you would not want to create 1000 web apps on the same farm since there is overhead for each IIS web site. 1000 site collections across one or a few web applications would have much less overhead.
2. Web apps can have different authentication mechanisms. For example if you wanted to use anonymous for the knowledge base but AD for HR benefits, it may make sense to separate them.
3. Web apps can have separate SSP (see above)
4. Web apps have separate content databases by default. You can force each site collection into its own content database, but it is a bit tricky.