Speculating about speculative fiction

I remember reading a piece by Heinlein lamenting the fact that "speculative fiction" lost out to "science fiction" as a general genre identifier. (And I believe he was writing in the early '60s, and it was already a battle fought and lost long ago.) IIRC, his main problem with the "science fiction" term is that it often caused him problems trying to sell stories that, while perfectly consistent with his other work, didn't have any of the elements people associated with "sci-fi" - no ray guns or rocket ships, not in the future, no aliens, etc. (And on the other hand, still "too weird" for the "mainstream.") It'd be nice if "speculative fiction" was the overarching term, I think. It's a shame we humans love putting things in boxes, but at least it's a bigger box.