Negative test cases

On a previous job interview loop, one of the people I spoke with gave me good feedback about the work I’ve been putting up on my Github repos. He observed at the time that I had been hitting the low-hanging fruit: i.e., the test cases dealing with good, expected data.

He was right. So I’ve gone back and updated my repos for the WordPress REST API tests to also include negative test cases, i.e., known bad data, to test the error behavior of the endpoints. So here’s what I focused on to do that.

First, several of the endpoints ask for IDs of this, that, and the other thing: post IDs, category IDs, etc. For those sorts of cases, I did these negative tests:

  • IDs that could be valid (i.e., were legit integers), but which did not actually exist as posts.
  • IDs that were specifically not valid, i.e., things that were not integers. E.g., “aaaaaaa”.
  • Using MAXINT as a post ID, just to use a GIGANTIC integer. Practically speaking I’d usually expect this to also be a “this post does not exist” scenario. But I’ve tested things in the past where throwing a thing a value slightly above a limit behaved, but throwing it a value WAY bigger than the limit did not. So I wanted to do this scenario too.
  • Also using MININT as a post ID. This is not only to test using a gigantic thing, but ALSO to test using a gigantic thing that could not actually be a valid post ID (i.e., because it’s a negative number).

Secondly, several of the rest of the endpoints didn’t use IDs, but rather, slug/tags. So for those endpoints, I did negative test cases that were very similar.

  • Slugs/tags that could be valid, i.e., strings, but which did not actually exist in the database (such as using “pancakes” for a category tag).
  • Slugs/tags that could not be valid. E.g., using “pancakes” with additional non-alphabetic characters.
  • Using MAXINT and MININT again.

In all of these cases, I threw the bad data at the endpoints and looked for specific error codes and error messages I was expecting to get in response. I also looked for 404 response codes.

I got the expected behavior by testing the various endpoints manually with bad data, in Postman, and seeing how they responded. That gave me the basic JSON response structure I would need to expect: that there would be both an error code and an error message, and an additional data object that would contain the response code within it. So my test cases looked for all three of these things.

As of this writing, I’ve implemented the negative test cases in Java, Python, and C#. Once I had them working in Java, it was reasonably easy to port them over to the other languages. In all three languages, this has now brought the test case count for the REST API suite from 20 up to 60.

In addition to learning the error behavior of the various endpoints, the most useful thing I’ve picked up on in this part of the project is how to get at MAXINT and MININT in the various languages.

In Java, it’s Integer.MAX_VALUE and Integer.MIN_VALUE. In Python, it’s sys.maxsize or -sys.maxsize–and this is Python 3, specifically. (Apparently, in Python 2, it was sys.maxint, but it changed; if I understand the change correctly, it was because Python 3 doesn’t have a hard and fast integer size limit, because it automatically recasts your number to a long on the fly if you go over the theoretical int limit.) And in C#, it’s int.MaxValue and int.MinValue.

Meanwhile, over in the Selenium test suites, there are fewer opportunities there for negative cases. But I did add some there that throw bad data at the search box, to verify that I get the appropriate messaging if there are no matching results. There’s some room for expansion there as well, to try to test the limits of what the search box can accept. So that’ll be the next place I add more negative cases.