Suppose that there is a completely sealed black box that you cannot touch or interact with in any way except by looking at it. And further let us suppose that a method for determining facts about the contents of the box drops your lap, either a deductive one, that starts from the information you have access to and produces claims about the contents of the box, or a hypothetical one, that you feed claims about the contents of the box and which it proceeds to either validate or refute. Or suppose that you don’t know how addition works, that you don’t know even whether adding two numbers is supposed to produce the number that counting two collections of objects of those cardinalities as a single collection results in. And, similarly, a method is given to you for adding two numbers together. How, in these cases, are we to evaluate the methods that we happen to come across? Obviously if we already had another method for determining the same facts we could compare the two, either formally by proving in some way that they must produce the same results, or by simply comparing their results in a large number of cases. But that is not the situation we find ourselves in here, here we are really in the dark. And indeed the study of anything must find itself in this situation at some point, because there must be a first method (because the history of people using methods to uncover these facts doesn’t extend infinitely into the past). So at least some real people have found themselves in this situation. How did they extricate themselves from it?
Now just because we don’t have a method for determining the answers we are pursuing in the general case doesn’t rule out the possibility that we know some specific facts by other means. For example, we might know that there are only five objects in the box, or that 2+3=5. Obviously if the method fails to agree with any of these specific cases then we know that it is a flawed method, at least in some situations. And since we have no way of determining its accuracy in any of the innumerable other cases which we know nothing about this may motivate us to set it aside as a way of knowing the answers we are interested in. But still, even if does not contradict these specific cases, this does not mean that it is correct in all cases, or even most of the time, although it does lend at least some credibility to that possibility. The number of cases we know about in other ways is obviously finite (given our assumptions), and possibly quite small in number, or even nonexistent. However, this doesn’t exhaust the knowledge we may already have about the subject of our investigations. Besides knowing specific facts we may also know certain general properties or bounds. For example, given the fact that we can observe the box from the outside we know that there are limits as to the size and volume of whatever is within it. And when it comes to addition we may know that certain laws hold, such as the associative law. Knowing such general limits allows us to check every claim made by the method, determining whether it conforms to these properties. Still, even if the method doesn’t violate these properties this doesn’t ensure that it is correct either. It is fully possible for the method we are handed about the box to still be in error about its contents, just in ways that don’t violate the size constraints. Similarly, there are many mathematical operations that are associative, and so just because the method obeys this property doesn’t mean that it is doing addition.
Such tests exhaust our ability to use what we already know to evaluate the method. But these are not the only resources available to us. There are ways of evaluating the methods simply by the results they produce, without appealing to any knowledge about what the results should be in the intended cases. One such way is by examining the coherence, consistency, and universality of the results. Examining the coherence of the results means seeing whether the claims made by the method can all “fit” together, or whether some contradict the others. For example, if our method tells us that there are five objects in the box in one application, that all the objects are cubes in another, and that there are four cubes in the box in a third, then we have a contradiction. Clearly not all of these claims can be true at once, and so we have discovered that at least one of them is false. This allows us to discover a flaw in the method without actually knowing which results are true and which are false, which is quite elegant. Of course it is possible that the contents of the box are in some kind of rapid flux, but we can, for the most part, rule this out by using the method over and seeing if it still produces those same results. Which brings me to consistency; we expect that the method will produce the same results with repeated applications. Of course in the case of the box there is the possibility that the contents may be changing, but we can reasonably expect that the contents are changing at some rate, and so that applying the method in rapid succession will result in answers that are closer to each other than applications that are farther apart in time. So while such inconsistencies aren’t definitive in the case of the box they may be suggestive. Arithmetic, on the other hand, we expect to be completely consistent (perhaps this is another one of those properties that we know about it beforehand), and so any inconsistency would seem to show that the method produces some errors. Finally, we have universality, which means that we expect the method to produce the same results for different people, assuming that it is used correctly by them. Again, the failure to be universal doesn’t definitively show the method to be failing in these cases, perhaps we are all working with slightly different information, but it is suggestive (especially since multiple people can use the method at the same time, which helps determine whether any inconsistency is caused by actual changes or by a failure of the method).
Again, even if the method in question is coherent, consistent, and universal this doesn’t prove that it is correct. Another way that we can test our unknown method is to see how it performs in situations other then where it was intended to be used. For example, we can use the method that is supposed to reveal the contents of the box to us on another box, one that we do have access to. There are three possible results of this test. If the method works flawlessly in these situations then it provides evidence, strong evidence, that the method is a good one. Or the method may simply fail to be applicable, the process by which it proceeds may simply not be possible when considering other boxes. Again, this lends some credibility to the idea that the method is a good one (although less credibility than working perfectly would), because it implies that the method is really proceeding on the basis of some connection to the subject matter that doesn’t exist in these cases. Finally, it may produce the wrong answers, which makes it less likely that the method is a good one, although it doesn’t rule out the possibility (it is possible that the method is simply very poorly designed). Obviously testing the method for addition in this way is a bit harder, because there is nothing that we could apply it to outside of numbers that it could produce a correct result for. Still, we might try “adding” a shoe and a bottle using it and seeing what happens. If the method produces any result at all this would make it quite questionable (since it is hard to conceive of methods that can work with both shoes and numbers without proceeding by simply by disregarding their contents; a way of adding that simply always yields the same result, which would have passed all our previous tests, may be revealed as questionable in this way). As a side note I would mention that I often test philosophical methods in this way, seeing what they would do with the question “what is water?” If they agree that water is H2O (producing the answer themselves, not by simply refusing to argue with science) or reject the question as not properly philosophical then they pass, otherwise I toss them out as flawed.
At this point we might also perform a sanity check (that’s a technical term) on the results of the method. Do we understand the claims produced, or are they completely opaque in meaning to us? Either of the methods could produce claims that contain words that are completely undefined, but pass all the other tests we have put them to. A method that produces consistent nonsense has no problem when applied to situations other than its intended application, because we have no way of determining whether that nonsense was right in those other situations, even if we know all the facts about them. For example, if we examine our safe using the method and it produces the claim that “gleeb is grue” we simply can’t tell whether to accept or reject that answer on the basis of the contents of the safe. I maintain that claims that are opaque in meaning are really pseudo-claims, they seem like they are claiming something but they really aren’t. Thus a method that produces them isn’t really producing claims at all, and hence isn’t really a method, which certainly warrants us to reject it as a method.
Finally, we can examine how the method itself works. When we pick apart the method we are looking to inspect where the answers come from. For example, in the method for addition we will find a set of rules for operating on two numbers. From the nature of these rules we can in turn deduce a number of general facts about addition, if the method is correct. Now in the case of addition this may or may not help us settle the matter. If we know certain facts about addition to begin with we may be able to prove the correctness of the method. But if we don’t then all this inspection will reveal to us is that it defines some mathematical operation. And in fact this, I would claim, is really all there is to the matter in this particular case, if we have decided so little about how addition works to begin with then there really is no fact of the matter yet about what addition is for us, and we can choose to simply accept this method as defining it. When it comes to the method for determining the contents of the box things are not so open. By inspecting the method we expect to find at least some possible way for information about what is in the box to make its way to us through the method. If the method doesn’t even contain the possibility of such a connection (if it produces its answers by looking them up in some specified table, for example) then we can conclude that it is not a method for discovering what is inside the box, and that if it is right it is only right because whoever dropped the method in our laps had some other way to determine the contents of the box and put that information into what we thought was a method. But usually such an inspection reveals at least a possible connection between the contents of the box and us. For example, the method may work by measuring air currents, which implies that the contents of the box affect the air around the box. Or it may proceed on the basis of intuition, implying that we have a mental connection to the contents of the box. Given these connections we can produce a judgment about the method based on what we already know about how information and knowledge works in general. Is it possible that the contents are affecting the air around the box, is it possible that we have a mental connection to its contents? While such considerations aren’t definitive, because this may be the case that shows our previous judgments about these issues to be in error, they are strongly suggestive.