In the early days of the web, CGI was indeed the only (practical) way to have dynamic content (you could do named pipes of files — and those were used in days before cgi, but that wasn’t practical at all).

CGI works by sticking a bunch of information in the environment of the process that is forked and then exec’ed (and possibly some in stdin) and then takes what comes out of stdout and spits that back to the requestor.

This doesn’t care one bit about what the implementation language is. Indeed, I wrote my early CGIs back in the day in C or C++. It was kind of painful. I later learned some perl in the early 90s and that was much less painful.

This works, up to a point. The problem is scale. Each CGI request is a fork and exec of a process. Thousands of requests means thousands of processes. That really doesn’t work well.

The solution to this is to remove forking and execing by either moving it into a thread in the web server itself, or dispatching the request to another process that handles the request without needing to fork and exec. mod_perl is one such tool to do this (a plugin moving perl into apache). Php (late 90s) also did this with implementing the language as a plugin in the web server itself rather than something that was forked and exceed. This became quite popular as it was perl-like (which was the early dominant web programming language) and could outperform perl cgis. There is still quite a bit of momentum from this period of time in the mid-90s — before the more enterprise-grade application servers started to take hold with more formalized languages behind them. If you dig around, you can find a lot of failed attempts in the late 90s to early 2000s too — languages and frameworks that just didn’t stick.

This brings us to the application servers where internal threads are spawned (or other approaches — this isn’t the case for everything) to handle requests rather than entire new processes — which can help with scale. As an external process this could be seen with FastCGI and then later became prevalent with other application servers. Note that with this the line between application server and web server got a bit blurry — many application servers could double as web servers, though weren’t optimized for handling static file IO in the way that traditional web servers are.

The generic application server has also paved the way to solutions where instead of a generic application server, you have the application itself either running an embedded web server or otherwise being the entire deployment. In such situations one doesn’t deploy a web application on an application server – it just is running itself and handling requests. Again, the goal of this model is to avoid the heavy price of launching new instances of the application and instead handle the requests inside the application with much lighter weight threads or similar approaches.

Here’s the thing though — all the solutions are deficient in some way, shape, or form. CGI, while easy has serious problems with scale. Plugins in the web servers get bound into the web server itself (apache vs nginx vs IIS vs …) and lose the common functionality of the language. Microsoft has its own parade of technologies it would like to promote. And if you know one language, wouldn’t you rather keep programming in it rather than have different languages in different parts of the stack (javascript in the client and Node.js)?

And so, you’ve got today. Some people work in a Java stack (with scala and clojure becoming not uncommon). Others in a C# stack. Others in a JavaScript stack. There’s quite a bit of php stacks out there. Lots of python. You can still find some perl stacks out there (and if you look at some low volume sites, you’ll still find CGIs). With cloud computing, Google has also promoted Go as a viable server side web language.

Each has its advantages, disadvantages, its frameworks and its servers. The relative popularity of these ebbs and flows as technologies around them change. They do different things well.

Yes, any general programming language can serve to write the server-side part of a web site.

However, the qualities of a programming language, in this subject as in other things, are usually only one of many factors that contribute to its popularity.

For example, I reckon that PHP became popular for websites because:

• It is extremely easy to upgrade from a static website to a PHP dynamic website – simply replace the file extension of your HTML file, place the <?php tag at the beginning, and, provided PHP is installed, you have a dynamic web site! The rest of the workflow is exactly the same as for a static website;
• Because of that ease of deployment, web hosts that were looking to propose dynamic web sites chose PHP, making it pretty quickly the most widely deployed server-side platform;
• It got into that market at the right time;

And once PHP was widely deployed, it became interesting to write more serious web applications in PHP in order to benefit from that wideness of deployment.

To say it in a more generic way: language adoption is often about the answers to these questions:

• How easy is it to do what I want to do?
• How widely supported is the language for what I want to do?

Am I right in thinking that a server just needs some kind of interface
such as CGI to make the server and the programming language work
together?

Almost. You need a web server that has some kind of software to allow it to respond to HTTP requests as well.

Think about how a static page is served. The server retrieves the HTTP request, finds the requested document from the filesystem based on the HTTP server’s configuration, and returns the static page.

CGI extends this concept by allowing you to designate a cgi-bin folder on the filesystem where executables or scripts can be stored. When you access a program via CGI, the HTTP server runs the process or script and passes the standard output back to the client rather than simply serving up the static document.

 If so then why are some programming languages (such as php) more popular than others?

The old CGI structure does not scale well over a large volume of requests. Different programming languages and frameworks for the web exist for different reasons, and each one does different things well. PHP is as popular as it is for historical reasons, as it was one of the first easy and cheap solutions for serving dynamic pages without resorting to CGI and had widespread hosting support. ASP was popular among Microsoft circles because it allowed VB developers to shift their skills to the web. ASP.NET (Web Forms) made it very easy for Windows Forms developers, many of whom were VB coders, to switch over to the web.

When a browser makes an HTTP request, it looks like this:

GET /search?q=cats HTTP/1.0
Host: www.google.com
Connection: close

… to which the server should send a response that looks like this:

HTTP/1.0 200 Success
Content-Type: text/html; charset=UTF-8
Content-Length: 1337

<!DOCTYPE html>
<html>
  <head><title>cats - Google Search</title>
  <body>
    <h1>About 415,000,000 results</h1>
    …
  </body>
</html>

Any code running on the server that listens for requests on a TCP socket, reads the request, and replies with the appropriate response will suffice. One dumb way is just to spit out a canned response to anyone who connects to TCP port 80, using a shell script:

$ nc -l 8000 <<'RESPONSE'
HTTP/1.0 200 Success
Content-Type: text/html; charset=UTF-8
Content-Length: 1337

<!DOCTYPE html>
<html>
  <head><title>cats - Google Search</title>
  <body>
    <h1>About 415,000,000 results</h1>
    …
  </body>
</html>
RESPONSE

Of course, that technique only barely appears to comply with the HTTP protocol.

A step up from that canned response is this simple Python program, which uses the http.server library in Python 3.

#!/usr/bin/python3

import http.server

class Handler(http.server.BaseHTTPRequestHandler):
    def do_GET(self):
        payload = '<!DOCTYPE html>... insert cats here ...'.encode('UTF-8')
        self.send_response(200)
        self.send_header('Content-Type', 'text/html; charset=UTF-8')
        self.send_header('Content-Length', len(payload))
        self.end_headers()
        self.wfile.write(payload)

http.server.HTTPServer(('', 80), Handler).serve_forever()

The HTTP server can be written in any language; that is just an example. Obviously, this example is very rudimentary. The payload is hard-coded — the program completely disregards the contents of the request — the URL, query string, Accept-Language header, etc. You could add code to generate meaningful responses based on the request, but then the code would get very complex. Besides, programmers would rather focus on writing the web application, not having to worry about the details of how to handle an HTTP request.

A more appropriate solution would be to use a web server, such as Apache HTTPD, IIS, or nginx. A web server is just a program that listens on the relevant TCP sockets, accepts multiple requests (possibly simultaneously), and decides how to generate a response based on the request URL, headers, and other rules. Ideally, many of the details, such as SSL, access control, and resource limits are taken care of via configuration rather than code. Much of the time, the web server will formulate a response that consists just of content from files in the filesystem.

For dynamic content, though, the web server can be configured to execute some code to generate the response. One mechanism for doing that is with CGI — the server sets some environment variables based on the request, executes a program, and copies its output to the TCP socket. A slightly more sophisticated solution would be to have a module that adds support to the web server for calling code in another programming language (e.g. mod_php for Apache). Yet another option is to write the web server in the same language as the web application, in which case the request dispatch is just a function call. That is the case with node.js and Java servlet engines such as Apache Tomcat.

The choice of technology is really up to you, and depends on the programming language you prefer to use, the hosting environment that is available to you, performance requirements, popular opinion, and passing fads. CGI, for example, has not been favoured lately, since the need to launch external programs limits scalability.

A web server is a program written in any programming language that handles “web traffic” over socket(s) adhering to standards/application level protocols (HTTP, etc). Most programming languages offer you to create a socket.

Am I right in thinking that a server just needs some kind of interface such as CGI to make the server and the programming language work together?

There is no need to have dedicated server program and your application program – they can be the same (disregarding any performance-related issues).

You can use some HTTP server library, e.g. libonion, even in your program coded in C (or C++, see also Wt). There also some HTTP client library (e.g. libcurl)

You can use other HTTP libraries, e.g. ocsigen & ocamlnet for OCaml.

There are several Web dedicated languages (outside of PHP), e.g. Opa, HOP, Kaya, etc… (both HOP & Opa can easily mix server-side and browser-side computations, but you have to do that painfully and manually in PHP, explicitly using AJAX techniques and hand-coding some Javascript for the browser. In contrast, HOP, Opa, Ocsigen are able to generate that Javascript).

You can also use FASTCGI technology to add some dynamic service to some web server … FASTCGI is better than plain old CGI which starts a new process for every incoming HTTP request, while a FASTCGI application can serve many HTTP requests in the same process. BTW, PHP can be configured to work as a FASTCGI application.

C.Queinnec observed that web browsing and continuations are significantly related.

PS. I don’t like PHP, and I believe that its popularity has historical and social reasons (not mainly technical ones). Indeed PHP was widespread well before AJAX became widely used, and is older than HOP or Opa (or Ocsigen).