Why do NPCS fight to the death in RPGs?

1. GMs (and the industry) moved away from morale rules because of the "roll vs. role" mentality which, in part, maintained that mechanics shouldn't govern character actions.

Thus, it became solely the GMs' purview.

(The other reason GMs abandoned morale rules was because they mechanically prompted all of the following stuff.)

2. Most GMs start by running dungeons. These are appealing because each room is firewalled from other rooms, making for a clear prep structure and easy running. (You don't have to worry about the whole scenario, just the current room.)

NPCs who run away break the firewall.

This becomes even more prevalent in the 4E-era My Perfect Encounter(TM) school of adventure design, when the game system itself would fall apart if monsters started running around willy-nilly.

To handle willy-nilly monsters cascading through your dungeon like free radicals, you need advanced techniques like an adversary roster: https://thealexandrian.net/wordpress/38547/roleplaying-games/the-art-of-the-key-part-4-adversary-rosters

And most GMs don't have that.

3. Taking monsters as prisoners ALSO breaks the firewall, because the PCs are going to want to question them about the dungeon.

This, again, requires the GM to break out of the current room and think about the entire dungeon as a whole. This is more difficult for the new GM.

4. Prisoners also create a logistical challenge which is perhaps interesting once or twice, but then quickly becomes boring.

5. The desire to avoid boring logistics will prompt players to solve the problem by murdering their prisoners. This is morally repugnant and, therefore, often undesirable.

Similar calculations will also motivate the PCs to shoot anyone running away in the back.

If you want a non-D&D example of the difficulty, check out THE RAID (2011). They try to take prisoners, but the logistics overwhelm them.

Once you move away from raid-type scenarios (which a typical dungeoncrawl is closely related to), bad guys running away are easier to handle and prisoner logistics (along with the tough choices accompanying them) usually become more interesting to explore.

But by that point, most GMs have already developed "fight to the death" as a habitual practice, so it tends to just kind of stick around.

6. On the flip-side, very few systems provide a viable system by which PCs can reliably flee combat. (Ironically, the original 1974 edition of D&D is one of the rare exceptions.)

Mechanically, this strongly incentivizes the PCs to fight to the bitter end, because...

...the alternative systemically boils down to begging the GM not to kill you.

(Players like to feel as if they're in control of their own destiny.)

7. This whole meme-sphere then feeds back into game design and scenario design, which (for example) balances combat encounters around the assumption that the bad guys are going to fight to the bitter end.

GMs who try to break away from that assumption...

...will find that the result is systemically unsatisfying: The PCs don't really feel challenged when the bad guys logically run for it, which leads to everything feeling like a cake-walk in which the last few rounds are just mopping up bad guys who aren't even fighting back.

No challenge. No satisfying conclusion.

=

Bad encounter.

Which, of course, encourages the GM to abandon the whole "running away" thing and leaving the bad guys engaged until the bitter (yet mechanically satisfying) end.