What happened in 2020 was truly rare: Not only was a President defeated running for re-election, but his party was defeated after holding the WH for just one term.
GHWB was the end of 3-term GOP run.
Ford finished off Nixon's 2 terms.
Hoover ended a 12-year run.
Carter is it.

And in Carter's case, there was a major primary challenge (Ted Kennedy) and a major 3rd party candidate (Anderson). The last time a President's party held the WH for one term was the Grover Cleveland-Benjamin Harrison era - D-R-D - but that's hard to fit in any pattern.

Last time before that is Zachary Taylor (succeeded upon death by Millard Fillmore) whose Whig Party only held the WH from 1849-53. The WH toggled back and forth then. Whigs (WHH/Tyler) from 1841-45 (barely "Whig" in Tyler's case). Dems from 1845-49. Whigs from 1849-53.

Needless to say, it was quite a rare feat to yank a unified Republican Party out of the White House after one term. Maybe we shouldn't be surprised at the downballot collateral damage to Dems since Biden clearly needed some GOP crossover votes to pull it off.