A knee-jerk answer would be prose quality. Yet plenty of bad writing can be found within the vaguely defined non-genre category named literary fiction. And so the question remains: why can it not be considered a genre itself?
That literary fiction should be the assumed ‘norm’ or standard measurement from which all other categories are regarded as deviations or inferior corruptions, calls to mind the philosopher Michel Foucault’s study of categories themselves. Any assumed norm in such thinking, in sexuality for example, or race, is always a blind spot, or ‘lacuna’, in such arrays – somehow special and above analysis itself. After all, it’s just normal (isn’t it?).
However, as we have seen in other disciplines, this is not the case. Any norm possesses its own set of conventions, boundaries and rules – just like any other category. In writing, one genre that displays this inclusion/exclusion policing in a very clear way is what has come to be called ‘historical fiction.’ Literary fiction valourises the contemporary, or, what one might term, the quality of coevalness. To write of any other age or time is to plunge into historical fiction. What is unclear is just where this boundary lies. Must it be under a year? Five years? Perhaps the point of no return is twelve years and three months. Who knows.
Another convention of literary fiction is that it valourizes interiority over exteriority. Its character’s mental and/or spiritual travels are held as more valid, or worthy of examination, than their physical travels. Questions of the interior life is this particular genre’s forté, if not its defining characteristic.
The third convention of literary fiction I will raise in the limited space available here is that obligatory moment of a character’s self-discovery that Joyce named the ‘epiphany’. It sometimes seems that any work of fiction much contain such a moment even to be considered literary. Indeed, reading a selection of literary fiction might convince anyone that such a moment is quite commonplace in life, when in fact it is merely one more of the reader’s expectations to be fulfilled. In literary fiction the main character is supposed to experience an epiphany (of some sort), while we the reader receive our escapist reward of imagining ourselves equally capable of any similar self-revelation. A capability that is so far out of sync with the real world that literary fiction asserts to document as to be, as I say, frankly escapist.
This brings up yet another convention of literary fiction, the conceit of ‘realism.’ However, this entry is intended more to open a discussion rather than to fully cover every possible permutation and so I will leave this particularly ridiculous rule of genre inclusion/exclusion for future consideration.