An open DNS server is a DNS server that responds to recursive queries (queries for domains that the DNS server is not authoritative for, such as websites that you go to, or domains that you send mail to, as opposed to your own domain), and does so for anyone (not just clients on your local network).
Originally, DNS servers and mail servers were all open. That's just how the Internet was intended to work. Over the years however, spammers started relaying through open relays, so the best practice became not to run open relay mail servers. For quite a few years now, best practice has been to avoid configuring DNS servers as both authoritative and caching (doing recursive lookups). Unfortunately, most DNS servers are still open.
The problem is that there are now DDoS attacks (attacks that send lots of data to a computer, so that it becomes overloaded) that use open DNS servers, using amplification (sending small packets to a computer that then sends large packets to the victim, making it possible to send more data to the victim). Specifically, a UDP DNS packet is sent with a forged source IP address (the one of the victim), and a query is made in a small packet (about 75 bytes) for a domain that has a very large response packet (using EDNS0, it can be 4,000 or more bytes). The response packet then goes to the victim. The victim gets about 50 times as much data as the attacker is sending out. So with a dialup connection, they could saturate a T1 line.

NOTE: These instructions show you how to completely disable recursion. This is the best practice. However, if you need to run a DNS server that is both authoritative and recursive/caching, you will need to check the DNS server documentation to find out how to enable recursive lookups only for your local network. It seems that there is no way to do this with Microsoft DNS; if so, you will need to use other DNS server software or use a hosted DNS service. If anyone is aware of a way to get Microsoft DNS to allow recursion only to specific IP ranges, please let us know -- lots of people would like to do that.

Fixing Microsoft DNS on Windows NT
•    Add HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\DNS\Parameters\NoRecursion with DWORD value 1 in registry
(NOTE: This may or may not work. Microsoft documents it in a way that both says it will disable recursion, and suggests
that it will not disable recursion).
Fixing Microsoft DNS on Windows 2000
•    Open DNS [Start->Programs->Admin Tools->DNS]
•    In the console tree, click the applicable DNS server.
•    On the Action menu, click Properties.
•    Click the Advanced tab.
•    In Server options, select the Disable recursion check box, and then click OK
Fixing Microsoft DNS on Windows 2003
•    Open DNS.
•    In the console tree, right-click the applicable DNS server, then click Properties.
•    Click the Advanced tab.
•    In Server options, select the Disable recursion check box, and then click OK.
Fixing Simple DNS Plus
•    Open Simple DNS Plus.
•    Go to the Tools menu and select Options.
•    Click 'Recursion' (under DNS) on the tree on the left side of the window.
•    Uncheck 'Perform DNS recursion'.
•    If you need to enable recursion for your local network, check that recursion box, select 'Only for the following client IP addresses', and enter the IP ranges of your network.
Fixing BIND
•    Open named.conf with a text editor
•    Use a line "recursion no;" in the "options" clause (or in the "view" clause)
•    If you need to enable recursion for your local network, you can use a "allow-recursion { ADD_LIST_OF_YOUR_IP_RANGES_HERE; }" line in the "options" section.
•    [Use caution; BIND files are easy to break]
•    For complete hardening, see