A creature’s initial distance to the PC’s is determined by the
GM according to circumstances. As a rule of thumb, use
2d6 times 10 feet for indoor encounters and 2d6 times 30
feet for outdoors.
"Surprise" in DFB is basically a perception check. In D&D 5e, there is no "Both partis surprised", I think. You only surprise the other side on purpose:
The GM determines who might be surprised. If neither side tries to be stealthy, they automatically notice each other. Otherwise, the GM compares the Dexterity (Stealth) checks of anyone hiding with the passive Wisdom (Perception) score of each creature on the opposing side. Any character or monster that doesn't notice a threat is surprised at the start of the encounter.
This is the way I usually play.
However, one thing I hadn't considered much is vision. If many monsters have "darkvision 60 ft.", for example, it is obvious that no one would actually see another creature at longer distances; maybe they'll hear a noise, etc., but they will hardly be able to discerne exactly WHERE it is coming from (unless they are in a narrow corridor, etc).
In fact, the idea that there is something in the dark just beyond the limits of your vision is so awesome that I am amazed that I don't remember ever using it!
What do you do? Do you press forward to find out? Does IT approaches, or hides away? How cool would it be if your heard a noise and then... there is nothing there!
So, initial encounter distance is basically always 60 feet, but you can hear some signs before that...
This alleviates MANY issues I have with 5e, including the lack of a clear rule for encounter distances and the excessive power of ranged weapons.
Notice that AD&D had a different rule, as mentioned by Delta:
"But on the other hand, OD&D says that monsters are seen at 20-80' (Vol-3, p. 9), and AD&D likewise says "A light source limits the encounter distance to twice the normal vision radius of the source" (DMG, p. 62) (as well as real-life experiments supporting the same thing, here; login required) -- so it seems like effective visibility is back to at least 60', equal to PC infravision."
So, in theory you COULD see a monster from beyond you infravision/darkvision... Maybe only a silhouette or shade? Or small eyes blinking in the darkness? Interesting ideas to consider!
There are also other issues to keep in mind.
First, a torch (or "light" cantrip) has only 40 feet of reach. This gives darkvision a clear edge. But the edge is not only the difference between 40 and 60 feet.
In fact, it is obvious that a torch in the darkness can be seen from a mile away, although the rules do not acknowledge this fact. (BTW, here is a video showing how USELESS a torch is in real life when indoors...)
Common sense would dictate that someone with a torch could NEVER surprise anyone else in the dark. Here is what I wrote in DFB:
Carrying a light in the darkness will ruin most PCs’ chances
of stealth, but other characters in the party might be able
to move around undetected.
The idea is that the paladin carries the torch while the thief sneaks around in the shadows, for example.
But all this talking assumes that monsters with darkvision are NOT carrying torches. Well, this is obviously what I would expect form OD&D (in its nightmarish underworld) or reality (such monsters would have no need for torches nor would put themselves in equal footing to diurnal foes), but I'm less certain in 5e, since even monsters with darkvision will get disadvantage on their perception checks in total darkness.
So torches would be USEFUL to them - specially on patrol duties!
And for someone carrying a torch, spotting a faint light in the distance might be more difficult than for someone in the dark.
Do notice, however, that light would be obvious for someone with darkvision in 5e; it would appear as shades of color in an otherwise black-and-white environment... which is also very cool.
Other senses - blindsight and truesight, for example - will generate different results. Likewise, a fog or thick foliage will alter expectations.
Finally, there is always the "turning a corner" issue. In a tight dungeon, with BOTH sides carrying torches, is is conceivable that they could fail to detect one another until they turned a corner.
And, if you're not careful, you can always be surprised when you open a door...
So, IN CONCLUSION... encounter distance should rely on light, darkness, the character's senses, circumstances and geography, instead of being determined randomly.
Here is how I describe "encounter distance" in my next book:
The GM will determine a monster’s initial distance to the PCs according to circumstances. For example, a monster might be found behind a door, or turning a corner in a dungeon. As a rule of thumb, use 2d6 times 10 feet for indoor encounters and 2d6 times 30 feet for outdoors. However, if the parties are meeting in a dark place, whoever has the better vision will gain the upper hand. For example, a monster with 60 feet of darkvision (or infravision) is unlikely to be surprised by a party that can only see 30 feet in the dark.
Needless to say, carrying a source of light (torch, etc.) in the darkness will make stealth near impossible for the bearer.
A smart monster with darkvision may inquire adventurers before entering their sight, and might attack first if they try to approach without giving answers.