Let's talk about genre.
We will start with a straightforward and conventional classification. First, take the umbrella of "speculative fiction". From Wikipedia:
Speculative fiction is a broad category of fiction encompassing genres with certain elements that are nonexistent in terms of reality, recorded history, or nature and the present universe, covering various themes in the context of the supernatural, futuristic, and many other imaginative topics. Under this umbrella category, the genres include, but are not limited to, science fiction, fantasy, horror, superhero fiction, alternate history, utopian and dystopian fiction, and supernatural fiction, as well as combinations thereof (e.g. science fantasy).This encompasses a lot; probably 98% of the existing RPGs, with the possible exception a few "crime" and "military" games.
Let's focus on three particular genres: fantasy, horror and sci-fi (also some of the most popular in RPGs).
"Fantasy" is in itself a big umbrella that could include urban fantasy, gothic, etc. Here, I'm focusing on the more traditional fairy tale-like stories which contains magic and magical creatures Tolkien, Dunsany, etc.
The same goes for horror; I'm focusing more on "speculative" horror here, not stories about murderers. But one thing is worth noticing. One of the important aspects of horror is that, more than other genres, it focuses on victims. This is not the case on "epic" tales which puts heroes against monsters, for example; here, the victims, if any, are unimportant. But some heroes fall victims to their own flaws. That is why the anti-hero a symbol of dark fantasy, even if the story is not focused on making you scared. What I'm saying is - there is some overlap between tragedy and horror.
Sci-fi should be more or less self-evident. This type of literature ponders about where technology could lead us (both physically and morally).
In the intersection between these three, we have this:
Fantasy + horror is dark fantasy. A Song of Ice and Fire, Elric, etc. - those are more fantasy than horror. Flawed heroes, innocents being victimized, and so on. "Scary monsters" are not the fundamental thing here. Notice the ASOIF's focus on victims of big events such as wars, poverty, etc., or Elric's focus on tragedy (and all the innocents he leaves in his way).
Sci-fi + horror is sci-fi horror. The movie Alien is sci-fi horror, but a lot more horror than sci-fi; as many have noticed, the spaceship looks like a haunted house instead.
Fantasy + sci-fi is science fantasy. Star Wars is a good example. A "sci-fi" movie that is really about a saving the princess from an evil king. "A long time ago in galaxy far far away" - like a fairy tale, not a perspective on the earth's future. It could probably encompass some highly speculative sci-fi where "sufficiently advanced technology is indistinguishable from magic" (read this, it is useful for our discussion).
As you can see, there are innumerable shades of gray. The lines were always blurred, since the beginning.
Edgar Allan Poe, for example, the great mystery/horror writer, dabbled in fantasy (The Devil in the Belfry is more satirical fantasy than horror) and science-fiction (the science of his time was about balloons, expeditions to the artic, and mesmerism). By the way, there are few monsters in Poe... except of the human kind. The real threat here is madness, resentment, malice.
Frankenstein is sci-fi horror. Dr. Jekyll... is he an alchemist or a scientist? Is there a difference?
The separation of "science" and "the supernatural" was not always as clear as we think it is today.
Weird fiction stands in the middle of it all. The classification is not exactly easy. See wikipedia again:
John Clute defines weird fiction as a "Term used loosely to describe Fantasy, Supernatural Fiction and Horror tales embodying transgressive material [sic]". China Miéville defines weird fiction thus: "Weird Fiction is usually, roughly, conceived of as a rather breathless and generically slippery macabre fiction, a dark fantastic ("horror" plus "fantasy") often featuring nontraditional alien monsters (thus plus "science fiction")." Discussing the "Old Weird Fiction" published in the late 19th and early 20th centuries, Jeffrey Andrew Weinstock says, "Old Weird fiction utilises elements of horror, science fiction and fantasy to showcase the impotence and insignificance of human beings within a much larger universe populated by often malign powers and forces that greatly exceed the human capacities to understand or control them." Jeff and Ann VanderMeer have also stated that weird fiction is a "mode" of literature, usually appearing within the horror fiction genre, rather than a separate genre of fiction in its own right.
I don't like any of those views on "weird fiction", to be honest. They seem reductionist or misleading. Despite having no credentials, I'll try to write my own.
So, the three greatest writers in the weird-pulp era were probably R. E. Howard, H P. Lovecraft and Clark Ashton Smith. They were probably some of the most influential to D&D, too (although C.A.S. is notably missing form the Appendix N).
Lovecraft is the only one who could be classified simply as horror (of the "weird" kind), with forays into fantasy and sci-fi. REH writes adventurous fantasy, despite some horror elements - his heroes are more Tarzan or John Carter than Dr. Jekyll or Viktor Frankenstein. CAS skillfully mixes the three genres, often with a humorous twist.
So, what do these three different authors have in common, in addition to being published in "Weird Tales"? Maybe it is the fact that they deliberately blur the lines not only between horror, fantasy and sci-fi, but also between science, magic and the unknown (and unknowable).
You can see it in HPL. The great "cosmic" entities are not simply aliens from outer space; their existence threaten the ways we view the universe and mankind, making science (and its separation for "the supernatural") meaningless.
You can see it in REH. Is the "Hyborian age" alternate history? Are the old myths of mankind based on alien beings such as Yag-kosha? What difference does it make, for Conan, if a monter comes from the bowels of earth or from outer space?
And you can see the same in CAS. When you enter the Vaults of Yoh Vombis, in Mars, you might as well be entering Hell and risking demonic possession (this story is somewhat reminiscent of the Alien movies, The Thing, etc.).
And you can see the same in the "new weird" authors. Ted Chiang and China Miéville, for example, extrapolate "science" from worlds in which the laws of physics are completely fantastical.
And with all this... We only scratched the surface.
There's a lot more to be said about the difference between epic and tragedy. And the possibility that (supernatural) fantasy and tragedy remained mostly separated in the modern era, until the gothic and the weird, is probably worth looking into (I'm thinking maybe fairies and dragons were not seem as "serious" matter during the era of Shakespeare, so they were relegated to comedy? I wouldn't know for sure). I wanted to write something about "gonzo" RPGs (weird + comedy), or how D&D was always "weird". Maybe there is something to be said about Sword and Sorcery, Sword and Planet, and...
But anyway, it's a start.