Mar 14 2008
Exercising Dynamic Web Frameworks, or, How Do I Test These Friggin’ Controllers Anyway?
One of the things that has always fascinated me about dynamic languages is how you go about testing them.
See, the trick is that when I write a closed implementation of some sort — whether it be a closed class or module — I’ve got some concept of the boundaries that my object’s state could be in. I’ve got a system which is locked in, probably with some state-machine-like semantics. So my unit tests wrap the flows that I understand, and verify that trying to violate those flows fails in some known way, and then I have reasonable confidence that things are good.
But dynamic languages that offer open implementations (like Perl, Groovy, and Ruby) really shoot that approach in the foot. Anything can be touched, and anything can be changed, and there’s nothing you (the original implementor) can do about it. This means that your unit test suite is going to be more incomplete, in the sense that your unit tests will exercise a much smaller portion of the field of possible ways in which your code might be used.
Worse, a common pattern in dynamic language development relies on external intervention in order to have it make sense. It’s extremely postmodern, actually: the code is gibberish without a context, making calls to methods and variables that simply don’t exist unless you have some kind of intervention. At this point, efforts to unit test are doomed to failure, because without the external intervention, there is nothing meaningful to test. With the external intervention, you’re really talking about an integration test.
Any interesting example of this is the particular case of controllers and views in Grails and Rails. Controllers are notoriously difficult to unit test, because they’re riddled with dynamically provided methods. Efforts to solve this via mocking are really treading on dangerous ground.
My solution right now is to not unit test them at all.
I used to be writing integration tests, which looked a lot like unit tests but fired up the framework before attempting to execute them. But that’s slow and dreary work, and I found it to generally be more effort than it was worth: inevitably, I ended up writing far more code to get to the particular unit test case than it took to do th implementation in the first place, and I often ended up mocking up so much that I was basically testing my mocking worked plus an if statement.
Enter functional tests. Before I start coding up a new controller, I start with writing functional tests (currently with Canoo WebTest through Grails WebTest Functional Testing) which encounter that controller and exercise the functionality that I want to see. This gets me all the guaranties that the integration testing provided, plus it provides a regression test suite that strongly resembles the user stories coming down the pipeline, and it exercises my views as well as my controllers.
Popularity: 7% [?]
There’s been a blog post rolling around my head for a while, every since this post on Raganwald, on the topic as to why I’d never use Ruby for a real project. Programs become brittle when you start having too many dependencies between modules- and dependencies work both ways, both depending upon some other module to change something, and depending upon all other modules to not change something. Ruby encourages- nay, requires- all modules to depend upon all other modules, because anything might change anything else. No code can ever be definitively declared “working” because some other piece of code might come along and, for example, change the core Symbol class.
Or, to put it another way, the only design Ruby allows is the big ball of mud.
“But if you think you’re free, try walkin into a deli and urinating on the cheese” goes the lyric. Some freedoms, obviously, need to be limited. Ruby is a free language, in the “Anarchy Burger” sense. Me, I vote for urine-free cheese.
Yeah. The trick in a dynamic language, I’m learning, is to stop trying to write code that is going to work in all possible cases, and to simply write code that you’re pretty sure is going to work in all the cases you’ve foreseen.
And, as part of this, I’ve learned that when I’m in a dynamic language, I need to relax my anal-retentive drive to push my code to not only work, but to not be able to be put into a situation where it won’t work. And, once I’ve started doing this, I’ve started to aim my testing at different targets: namely, at these routes through my web app instead of at the code itself.
For the MVC side of things, where code reuse (in your “Second Derivative of Programming” sense) is basically zero anyway, it’s actually not a bad approach.
In other words, all you can write in Ruby is throw away code. Code that isn’t correct in any sort of general sense, but is only “correct” in the narrowest possible sense- that it seems to more or less kind of work in the way it’s being used at the moment, but which is only one minor change in some other unrelated peice of code from breaking.
This means that Ruby is a mayfly language. It’s popular now, but five, ten, twenty years from now it’ll be non-existant. The only way to fix a big ball of mud is to rewrite it- and it’s throw away code anyways, just throw it away and rewrite it. In some other language. It won’t even leave legacy code around, like Fortran and Cobol. Is ugly and painfull as Fortran-77 or Cobol code is, it at least still works. Ruby will be more like APL or PL/1. Languages that collapsed so badly all projects written in those languages had to be rewritten in other languages, they couldn’t be maintained.
Reganwald framed this trait of Ruby as an advantage- that Ruby doesn’t need to go through a bureaucratic and ponderous standardization process to evolve. But there’s an advantage to having a bureaucratic and ponderous standardization process to evolve. The one huge advantage Jave has that everyone agrees on, is the huge body of working code, especially reusable libraries, written in Java. New features can be added,but code written for Java 1.1 and 1.2 a decade ago still works.
However slow, ponderous, and painfull development in Java is, it’s still possible to make forward progress. And all the productivity advantages of Ruby are illusory.
This isn’t to say that there aren’t things I’d do in Ruby. I do write a fair bit of throw-away code- code that almost certainly won’t be in use five years from now. That’s OK. But not all the code I write is throw-away code, and I’m not ever going to write to code that I may not want to throw away in Ruby. Because I’ll have to throw it away anyways.
Notably, there’s a lot of throw-away code in Java — practically all that stuff built on top of Struts is simply being rewritten in Spring Web Flow, Grails, or some enterprisey framework. It’s actually the light side of The Big Rewrite: there gets to be a point where business processes change so much that a whole new application is called for, and you can update the system in the process.
So throw-away code isn’t unique to Ruby — it’s endemic to the whole way of doing web application development right now. I blame the fact that web app development used to be done in Perl, which PHP was an improvement over, and Rails is a further improvement on that. The idea of doing things differently (e.g. 1, 2) has pretty much been ignored.
This damning condemnation of obsolescence aside, it’d be nice if you could salvage something from that previous implementation, and there’s a surprising amount salvageable from those venerable Struts apps, assuming you’re moving to some Java-based technology. Another major save is for people who have been doing SOA — that provides nice firewalls for the big rewrite change, and it’s proved itself out a lot more than its buzzword status would imply.
Note that some bonus for the functional testing is that they’re implementation-independent. I can just as easily fire them at an entirely different implementation of the website, and get a guaranty that the same flows work through. Which means you’re at least capable of carrying the specs on past the big re-write.
I think there is value in writing both functional tests (e.g. with WebTest or Selenium) AND unit tests for classes such as Grails controllers. I agree that currently it’s a little painful to do so, but just because it’s painful doesn’t mean it’s not valuable.
Using mocking allows me to quickly unit test controllers, especially with recent tips on how to do this using ExpandoMetaClass (http://blogs.bytecode.com.au/glen/2008/03/12/mockfor-march—unit-testing-grails-controllers.html) and run the tests right inside of IntelliJ. You’re right, it won’t catch the case where a property name changes and therefore a dynamic method changes, but it does help you make sure that any logic you have in your controller is being properly followed. I also find that unit testing helps me think more about the problem I’m trying to solve as well as put things in the right place (i.e. put business logic in the service or domain class, rather than in the controller).
Even with near 100% code coverage on my unit tests, there are times when my functional tests fail. Oh, and guess what - the reverse is true as well. Functional and unit tests work well when they’re used hand in hand.
As for the big re-write, I have rarely been involved with a case where functional tests from the original application could be applied to the new one being built. Things simply change too much, even if it’s a strict re-write.
@Mike
The question boils down to cost/benefit analysis. Considering that my controllers tend to be three to four lines in length — at most — I’m just not sure the mocked tests are worth it. And, in the spirit of dynamicism, it’s fair to ask why I would bother testing for a model value if it’s never actually used in a page? And if it’s used in a page, it’s fair game for a functional test.
The fact that I’m not unit testing the controllers means that I don’t put a lot of logic there. I’m the kind of person who feels spooked by writing any non-trivial code without tests, because I know that I am my own coding horror. It’s probably not the greatest idea to bank on my fear overpowering my laziness, but it seems to be working out okay for me so far.
As for porting the functional tests forward — I’m about to port an old Rails 1.2 site to Grails, and I have every intent of moving forward the functional tests. :)
Brian:
Nice work, leaping from “you can write a big ball of mud in Ruby” to “since you can’t stop people from writing a big ball of mud, all RUby programs are big balls of mud.”
The interesting thing is, I agree with the premise that you can’t stop people from writing big balls of mud AND with the premise that some or many Ruby programs will eventually decay into big balls of mud. But sadly, my belief about Ruby programs decaying into big balss of mud comes from my experience with FORTRAN, C. C++, Pascal, COBOL, and Java programs decaying into big balls of mud.
I haven’t seen a programming language where you can prevent people from writing big balls of mud with interdependencies, nor do I believe that sealing classes and/or modules will prevent people from writing big balls of mud with interdependencies. They will just be dierent kind sof big balls of mud.
I think that a desire to use programming language features to affect cultural change is laudable, so by all means keep evangelizing your intuition.
Please post your thoughts as a blog post so that I share a del.icio.us link to them.
Brian:
Nice work, leaping from “you can write a big ball of mud in Ruby” to “since you can’t stop people from writing a big ball of mud, all Ruby programs are big balls of mud.”
The interesting thing is, I agree with the premise that you can’t stop people from writing big balls of mud AND with the premise that some or many Ruby programs will eventually decay into big balls of mud. But sadly, my belief about Ruby programs decaying into big balss of mud comes from my experience with FORTRAN, C. C++, Pascal, COBOL, and Java programs decaying into big balls of mud.
I haven’t seen a programming language where you can prevent people from writing big balls of mud with interdependencies, nor do I believe that sealing classes and/or modules will prevent people from writing big balls of mud with interdependencies. They will just be different kinds of big balls of mud.
I think that a desire to use programming language features to affect cultural change is laudable, so by all means keep evangelizing your intuition.
Please post your thoughts as a blog post so that I share a del.icio.us link to them.
One thing occured to me as I was reading this post: if you have code that’s designed to be reusable or will be used once by someone else, then having a set of unit tests that convey how the library is supposed to work makes it easier for someone to change the underlying system. Without the tests, it could become quite difficult to tell whether making a change of that magnitude is going to break libraries that are being used. If the library author provides tests that detail their vision of how that library works, then the person making the change can make the change and run the tests again to see the effects and then determine if they can accept the behavior as is, or if they need to fix their change, make additional changes, or rollback their change and find a different path.
In a lot of ways, i think testing at the unit level for dynamic languages is much more about “change detection” than it is about “correctness”. If you have tests at the application level that ensure your application as a whole works correctly and tests that determine when the library you’re using is doing different things than expected, you’ve got a lot of the same benefits as unit testing in static languages (though it may sometimes require more investigation to determine what caused the change in behavior in a dynamic language.)
@Andy
I totally agree that unit tests for libraries are critical. Unit tests for business logic (including services and domain objects in MVC) are really necessary. But, since most controller methods consist of “call this service/domain method with the argument in the ‘id’ parameter, and then render”, it seems like the pain ratio of it is pretty high.
This situation only gets worse when you consider how frequently controllers change. They will often start pulling in new pieces, adding in new pieces to the model, change their action names, etc., etc., etc. Each of these changes requires an updating to the API, which then requires changes to the unit tests, and since building and changing the unit tests isn’t exactly easy, I’m really not sure it’s worth my time when I’ve got the functional tests already at my back.
There is a major difference between the library problem space and the controller problem space, so the two aren’t really comparable when you’re thinking about best practices. Controllers are exceedingly fluid, changing often and having no intended user other than the views which call them. The value of having a consistent API is extremely little. Libraries, on the other hand, provide an established API with lots of potentially unknown user code, so the consistent API is critical.
@Reg
I think there’s a problem which you’re glossing over. Sure, you can write a Java program that’s a ball of mud, but it takes a lot more effort than to write a Ruby ball of mud.
Particularly when you factor in “hidden dependencies” like external mangling of the classes (e.g. Logger, Symbol), it becomes very hard very quickly to see what all the dependencies are for a given piece of code. At least in Java, I can lock down my code and know that it’s going to work on any JVM, no matter what other code has been executed beforehand. That’s not something I’ve got in Ruby — that kind of code safety is simply not supported by the language.
“you can write a Java program that’s a ball of mud, but it takes a lot more effort than to write a Ruby ball of mud”
I think we are bandying unsubstantiated opinions around. I find it very hard to believe that any programming language gracefully guides programmers to creating programs with well-separated concerns.
My conjecture (and I admit it is a conjecture) is that if (If!!!) you were to examine a bunch of actual projects in language X and another bunch of actual projects in language Y and you found that X had more BBoMs than Y, the cultures of X and Y would have a much bigger impact than the features of X and Y.
But I can tell you from observation that if it is difficult to write a BBoM in Java, the Java programmers I have met have proven their prowess :-)
Wow. We’ve acronymed “big ball of mud”. Does that make it an industry-standard term?
You’re right in that my second sentence in that post was an unsubstantiated conjecture. What about the hidden dependencies assertion? That’s an actual example of a difference between Java and Ruby which provides a clear protection against code interdependency.
I didn’t invent the BBoM acronym, I have seen it for a while. I’m sure you’ve read:
http://www.laputan.org/mud/
The shared dependencies issue with Ruby is well-known. It actually reminds me of “DLL Hell” in Windows. If you think of modified classes as versions, you have a version control problem. OTOH, you also have an opportunity to reach out and modify other people’s code without their permission, which is a no-no to some people and a blessing to others.
For example, you may want to modify a base class and make sure that another piece of code you didn’t write uses your modified class. For example, you could wrap some AOP around various method calls and what-not.
Is that a worthwhile trade-off? I don’t know yet. I know that Java takes a very strong position on this: not only can you not modify core classes without magic (I speak as a magician, I have written code that uses a custom class loader to modify the byte codes of classes on the fly), but you also cannot subclass certain core classes like String.
That has been enormously frustrating for certain work I have done, but the powers-that-be have decided that they want to turn the knob over to “safe.” Matz has explicitly said that Ruby is a pair of scissors and Ruby programmers are encouraged to run with them.
Good idea? Bad idea? I honestly do not think the matter can be decided by simple inspection and reasoning. My experience with open classes in Ruby so far is mixed. There are things I like and things that concern me. My experience with Java so far is that I dislike its locked down approach. I am interested in how languages liek Scala are approaching these issues.
BTW, this problem is much bigger than just Open Classes. I recall an exchange of blog posts over final classes and final methods. Declaring that nobody can subclass your class is really very similar to declaring that nobody can open your class.
Can you describe situations when this is enormously frustrating in the earlier case? I’ve never bumped into a problem and went, “Boy, y’know what I need to do? I need to globally mess with the way this class works.” The closest I’ve come is appreciating some features of Rails inflections, but that’s nothing a set of utility functions wouldn’t solve. Actually, the whole enterprise makes me appreciate Ocaml’s module approach a lot more — this is kinda funny, because I really hated on the module approach when I first picked up Ocaml.
I suppose, at the end of the day, I’m just not a fan of running with scissors into production.
“Can you describe situations when this is enormously frustrating in the earlier case? I’ve never bumped into a problem and went, ‘Boy, y’know what I need to do? I need to globally mess with the way this class works.’”
In Java, not open a class but subclass a class. For example, it would be nice to use the type checking system to enforce certain encoding issues. Let’s use a trivial case: to have two kinds of String, HTMLString where HTML entities are already converted and PlainString where they are not.
if you can subclass String, you can use Java’s strong, static type checking to ensure that you do not accidentally pass a PlainString to a method expecting an HTMLString. But since you cannot subclass String, nor is it an interface you can implement, you must write a container class and constantly reference and dereference an instance member.
Some folks argue this is a good thing, because if they write a method expecting a String, they want a guarantee it is a String and not something else I have deemed to have Liskov equivalence to a String.
p.s. Of course, if I could rewrite Java history, String would be an interface. But then again, most Java programmers are class-oriented, not interface-oriented.
I actually had that problem with my first attempt at an open source piece of software. I implemented BigInteger and BigDecimal using GMP, but there was no concept of “structurally equivalent, but not inheriting from”.
As far as I can tell, the whole nominal typing thing is a failure. Inheritance is simply fraught with problems which make it extremely subtly wrong in many, many cases. There’s still a lot to be said for the correctness guaranties of static typing, but making all of that nominal and not structural is just wrong.
BTW, String isn’t too far from an interface these days — check out CharSequence that came in with 1.5. Of course, people still accept “String”s all the time when they could perfectly well expect “CharSequence”s, so the kind of thing where Sun is paddling upstream. But, if you hit a small piece of code or are working on a library and you want to be really nice to future maintainers, there you go.
I can appreciate the argument that an open language encourages BBoMs. But, in a weird way, I think closed languages may also actively promote BBoMs. One of the benefits of a statically typed languages is that tool support is much easier to add because it’s easier to reason about the structure of the code. Java has given rise to amazing tools like Eclipse and IDEA, which makes traversing large, dependency riddled code bases simple. In Java, a minor BBoM isn’t that tough to work with. But at some point your tool may lull you into creating a big mess. So when you say “Culture X” may promote BBoMs, I read that to say both “the culture of Java+Eclipse” or “the culture of Ruby+Open Classes”. In the end it seems that some person has to be fanatical about dependency management to divert the disaster, regardless of the platform.
[...] Exercising Dynamic Web Frameworks, or, How Do I Test These Friggin’ Controllers Anyway? [...]