There are two commonly used methods for calculating the number of fish to keep in your tank. The first considers the amount of available oxygen introduced by the surface area of your water, while the other is a simple formula for assessing the total length of fish per volume of water in your tank.
The formulas for saltwater and freshwater differ, because saltwater is denser.
Surface Area Method
1" of large deep-bodied fish (excluding the tail) per 20 square inches of water surface area. For example, a 3" long, mature Catfish would require 60 (20 x 3) square inches of water surface all to herself. The tank would have to be at least 12" long (left to right) and 5" wide (front to back), or at least some other combination, like 12 x 5 or 10 x 6, which equals 60.
1" of large deep-bodied fish (excluding the tail) per 60 square inches of water surface area. For example, a 3" long Damselfish would require 180 (60 x 3) square inches of water surface all to herself. The tank would have to be at least 18" long (left to right) and 10" wide (front to back), or at least some other combination, like 18 x 10 or 15 x 12, which equals 180.
Keep in mind, these calculations are guidelines based upon the traditional-shaped rectangular aquarium. If you have an unusual-shaped tank, like the popular hexagon, the number of fish allowed may differ due to a different surface area.
1" of slim-bodied fish (excluding the tail) per gallon of water; so a 2' Catfish would need at least a 24-gallon tank all to herself.
4" of small to medium fish (excluding the tail) per 10 gallons of water; so two 2" less aggressive Catalina Gobies could live together in at least a 10-gallon tank.
2" of larger, fast-growing fish (excluding the tail) per 10 gallons of water; so a 20" Polka-Dot Grouper needs at least a 100-gallon tank to herself.
These calculations, or rules of thumb, don't consider the your experience as an aquarist or your aquarium's filtration system. As always, it is important to maintain excellent water quality through efficient filtration and periodic testing. Increasing or improving the aquarium filtration used may also allow you to increase the number of fish that can be kept in your system.
Additionally, remember to base your calculations on the full, adult size of the fish. Because of this, it's a good idea to stock your marine aquarium at 75% of it's total capacity to allow for growth.