Assuming that a philosophical theory does not suffer from any logical or methodological flaws in the argument for it there are basically four ways to argue for its rejection. It can be argued that the theory does not actually accomplish what it sets out to do; that it raises further questions, leaves some things unexplained, or is in some other way incomplete; that some opposing theory provides a better explanation; or that the theory itself implies a contradiction or something equally absurd. The last of these ways, called the reductio ad absurdum, will be my topic here.
We can divide arguments by this method into two categories, those which argue against a theory by showing that it implies something extremely absurd and unintuitive, and those which derive a formal contradiction from the theory, for example by showing that an ethical theory recommends opposing actions in the same situation. Obviously the first kind is a much weaker counter-argument. Certainly absurd things can be true, despite their absurdity. Thus a carefully crafted philosophical theory, that explains everything it set out to explain perfectly, could not be rejected just because it makes an apparently unintuitive claims. There are, however, two kinds of philosophical theories that such a reductio is problematic for.
The first is those theories that rest on supposedly intuitive premises. If the theory is supported only intuitively then showing that it implies something unintuitive takes away that support. Because if we can accept this unintuitive conclusion then we can equally well accept the negation of one of the premises it rests on, and thus the fact that its premises are intuitive gives us no particular reason to believe it. This problem can possibly be escaped by arguing that the premises are much more intuitive than the derived absurdity is unintuitive, and thus that it is better to accept the absurdity rather than to reject one of the premises. But this is a weak response, because the strength of intuitions varies from person to person, and it is obvious that by making this response the proponent of the theory is grasping at straws.
The second are those theories that make normative recommendations (ethics and epistemology usually). The special case we are considering here is when it can be shown that a theory in one of these areas says we should do something (that something is ethical or is knowledge) when it intuitively seems that it is not. Because these are normative disciplines it would seem that evolution and experience should have shaped our intuitive judgments in these areas to conform fairly closely to the correct judgments, since making the wrong judgment in normative matters is to our disadvantage. Of course our intuitions in these areas can still be wrong, but here it is reasonable to give them the benefit of the doubt. So when a theory about normative matters contradicts our intuitions it is not unreasonable to demand an explanation as to why our intuitions are wrong in this case, and to reject the theory if such an explanation cannot be given. (But usually giving such an explanation isn’t too hard; it is often possible to show that our intuitions on matter arise from an extrapolation from certain similar and more common situations in which they are correct.)
But, except in the special cases mentioned above, showing that a philosophical theory implies something absurd is not enough. It is better then to show that it implies a contradiction. Since contradictions are impossible the theory is thus at best a flawed description of the world. We can further sub-divide contradictions into two kinds, external and internal contradictions.
External contradictions are those that can be shown to arise between a philosophical theory and something else. Most commonly such contradictions are between the claims of one philosophical theory and another (about a different subject obviously, contradicting competing theories is to be expected). Unfortunately in this case the existence of such contradictions isn’t itself sufficient to form a counterargument to the first theory. It could be that the other theory it is in contradiction with that is in error, and that the theory we are arguing against is indeed the correct theory. However such contradictions can be made into a relatively compelling argument if the theory in question conflicts with numerous other theories (because it is hard to believe that all of those other theories are in error), or with theories that have extremely wide acceptance. In rarer cases it is possible to show that the philosophical theory is in contradiction with science or experience (certain philosophical views of time and change, for example, are incompatible with general relativity). In such cases the philosophical theory is thus clearly defeated, because evidence provides stronger justification than any amount of analysis and argument to the best explanation ever can. But such cases are very rare.
But even better than external contradictions, as far as arguing against a philosophical theory goes, are internal contradictions. An internal contradiction is when we can derive claims from the theory that are mutually exclusive. Of course such contradictions are rarely in plain sight, usually they must be derived from separate parts of the theory. Which of course leaves some wiggle room for the proponents of the theory to argue that the derivation of contradictions is in error. But still, when such contradictions can be derived they do force a revision of the theory, there is no way to bite the bullet and accept internal contradictions without throwing out the very methods with which the theory has been constructed.
Now all of this may make the reductio ad absurdum seem like an attractive way to argue against a philosophical theory. It makes use only of the theory under scrutiny, and so there is no need to worry that possible criticisms of other theories by its proponents can help it out (as they could when trying to show that some other theory is superior to it). And when an absurdity or contradiction can be shown to exist there is little to do except revise the theory that led to it. The problem is that it is very easy to revise a theory so as to avoid contradiction (and possibly absurdity). Below I have diagrammed the structure of an argument for some hypothetical philosophical theory, and a contradiction that follows from the same premises (the contradiction could of course be derived solely from within the claims of the theory, but the same things can be said about such a situation). Along with it I have illustrated two possible ways to respond to that contradiction by revising the theory.
In response A the parts of the theory that gave rise to the contradiction are dropped, and the parts of the theory that were supported by the removed parts are now made into premises. In response B the offending parts of the theory are again removed, but this time they are instead replaced with weaker versions, which lend support to the propositions that were supported previously by the troublesome parts, and which are now too weak to derive the necessary contradiction. Such alterations are very easy to construct in response to any contradiction. Thus contradictions will generally be unsuccessful in convincing people that a theory should be rejected, and will convince them instead that it just needs some slight modifications. Of course such slight modifications in response to problems are extremely ad-hoc, and ad-hoc modifications are a sign that there is something wrong with the theory. However, such reasoning rarely convinces people to drop a philosophical theory.
Thus the reductio ad absurdum is a relatively poor way to argue against a philosophical theory, at least by itself. But sometimes it can be useful in forcing a particular premise to be rejected, or the theory to be modified in some other way, and then arguing against the modified way (perhaps the modifications leave it open to some other kind of criticism), so it is not something we should try to do without either.