Well, for starters to me, fantasy usually always takes place in a secondary world during (preferably) the medieval period where a systematized supernatural tie occurs. There are exceptions to this, obviously, and borderline cases such as Sandman and The Dark Tower, but to me this is usually the criteria. As for Sandman, it mostly takes place in our world (various centuries even), in dreams, Faerie, Hell, The Dreaming, you name it. But all those planes are still centered on our world. It doesn't focus on the organized otherness of the secondary worlds in the way fantasy usually does. Most of the inner workings of the odd stuff is never really explained (or sometimes it’s just borrowed from various mythologies). The strangeness is chaotic, as it would be in an absurdist play or maybe a David Lynch movie. Is Lynch a fantasy film maker then? Gaiman is just a talented author who uses the various troupes that are present in most fantasy and sci-fi stories for his own ends and in doing so, creates a nice illusion of familiarity while serving us something fresh. But I don't think that it makes him a fantasy author in the traditional sense.
So I guess what I am saying is, that to me, fantasy has to have at least some swords and sorcery. But it doesn't have to be a bad thing. Have you read George R. R. Martin? I mean, in the end, all that matters are the story-telling and the plausibility of the situations and characters. It can be a parody, a coming of age tale (which is the most command and also the most boring formula) an apocalypse tale, I do not care. All that matters are whether it's good. And that is a totally different story.