MongoDB repair on bad shutdown

I've been working on a little side project and I've been using MongoDB.  One insanely annoying thing I've run into a couple times is if Mongo doesn't shutdown cleanly then you have to repair it. The next time you try to start it up you get errors.  You'd hope that these things would happen automagically but oh well...I'm just starting to learn it so I may be missing out on the larger reason.

You may end up seeing something in the console like this when you try to connect to your local server,

:~$ mongo
MongoDB shell version: 1.8.2
connecting to: test
Tue Jul 26 20:45:17 Error: couldn't connect to server 127.0.0.1 shell/mongo.js:79
exception: connect failed

or this (this one was just me putzing around trying to get it going),

:~$ sudo mongod
mongod --help for help and startup options
Tue Jul 26 20:47:35 [initandlisten] MongoDB starting : pid=3358 port=27017 dbpath=/data/db/ 32-bit
** NOTE: when using MongoDB 32 bit, you are limited to about 2 gigabytes of data
**       see http://blog.mongodb.org/post/137788967/32-bit-limitations
**       with --dur, the limit is lower
Tue Jul 26 20:47:35 [initandlisten] db version v1.8.2, pdfile version 4.5
Tue Jul 26 20:47:35 [initandlisten] git version: 433bbaa14aaba6860da15bd4de8edf600f56501b
Tue Jul 26 20:47:35 [initandlisten] build sys info: Linux bs-linux32.10gen.cc 2.6.21.7-2.fc8xen #1 SMP Fri Feb 15 12:39:36 EST 2008 i686 BOOST_LIB_VERSION=1_37
Tue Jul 26 20:47:35 [initandlisten] exception in initAndListen std::exception: dbpath (/data/db/) does not exist, terminating
Tue Jul 26 20:47:35 dbexit:
Tue Jul 26 20:47:35 [initandlisten] shutdown: going to close listening sockets...
Tue Jul 26 20:47:35 [initandlisten] shutdown: going to flush diaglog...
Tue Jul 26 20:47:35 [initandlisten] shutdown: going to close sockets...
Tue Jul 26 20:47:35 [initandlisten] shutdown: waiting for fs preallocator...
Tue Jul 26 20:47:35 [initandlisten] shutdown: closing all files...
Tue Jul 26 20:47:35 closeAllFiles() finished
Tue Jul 26 20:47:35 dbexit: really exiting now

I found the answer in the comments of this blog.  Run the following commands and all will be well,

sudo rm /var/lib/mongodb/mongod.lock
sudo chown -R mongodb:mongodb /var/lib/mongodb/
sudo -u mongodb mongod -f /etc/mongodb.conf --repair
sudo service mongodb start

 HUZZAH!

The Goal

The only right way to write code is to use TDD!

Scrum should only take 5 minutes!!

You should ALWAYS use DDD to develop your model!!!

NOSQL is the only way!  Down with SQL!


That's the sentiment you can find in a lot of arguments between developers.  It's too black and white, too absolute.  The issue is that the focus leaves the goal.  It artificially limits the solutions you can come up with.

In the case of TDD, you can most certainly write code before tests.  What do all software development practices boil down to?  Building easily maintained software.  It isn't about creating beautiful code, it isn't about designing a family of classes, it isn't about creating a model that sounds like a spoken language.  Building easily maintained software can be done in any number of ways.  Something like TDD helps to reinforce proven practices for maintainable software but it's not the only solution.  I'm a TDD practitioner but you know what?  I've got no beef with writing code first.  You don't always need to TDD.

In the case of agile development, for anyone adopting agile, there's a LOT of focus on the religious practices that come with it.  You will, for a long time, follow these practices blindly, even when they don't improve your process and become a nuisance.  The answer is that not all practices may make sense.  Take a step back and look at what they are intending to achieve.  A 5 minute scrum doesn't mean you're doing a good scrum.  You may find that, for your group, have something like 8-10 minutes is more comfortable.  Don't hate yourself because you're not following it to a T.  You've done exactly what these practices were intended to do; maximize your productivity.  That's the goal of this whole thing.

In the case of NOSQL, it's a poor musician to blame his instrument.  People finally realized that they were using SQL as the hammer to nail everyone's persistence problems.  With the host of new applications we build, SQL isn't always the answer.  Maybe it's a document database.  Maybe it's a key-value store.  Maybe you need something that natively supports object graphs.  Instead of recognizing the fact that applications and their persistence mechanisms come in all shapes and sizes, the sentiment has become that SQL is the scourge of software development.  We have ourselves to blame on that one, trying to force an object model into a system that poorly represented it.  Somehow this is SQLs fault?  We stopped questioning why we did the things we did.  Take a step back, ask yourself what you're really trying to do and then it becomes a no-brainer.  That's what gave birth to all these new forms of persistence.

For some, it may take a while to see it but you'll get there.  Don't fall into the trap of blindly doing things without re-evaluating constantly.  There's more than one way to do something and you shouldn't let software purity or zealotry prevent you from considering it.  It leads to the innovations that improve our lives as developers and the people who use our software.

Thoughts on the Test Pyramid

About a half year ago, one of my coworkers sparked a discussion on the test pyramid.  Specifically, he referenced a link and asked our thoughts on it.  Since I like to geek out, I had to respond.  I actually spent some time creating what I thought was a reasonable analogy so I figured I'd share it.  I'm also offering this now as I have another post I really want to discuss in regards to the boundaries in testing between a QA group and developers.  Is there one?  Should there be one?  How do we deal with test overlap?  It's something I've been mulling around and I need to think out loud.

So, here's how the original thread went,

S: 

Some interesting posts on testing pyramids and automation. I'm a proponent of using this pyramid metaphor for organizing our test strategy. Thoughts?
http://benhutchison.wordpress.com/2008/03/20/automated-testing-2-the-test-pyramid/
http://blog.mountaingoatsoftware.com/the-forgotten-layer-of-the-test-automation-pyramid

Me:

It's an effective way to describe a proven testing strategy. The simple law here is that tests are all about feedback and how valuable that feedback is. The value is tied directly to their maintenance cost. If you look at it this way, you stop thinking in terms of layers. For example, brittle unit tests are high cost and thus less valuable than verifiable, repeatable, solid unit tests. This is the mindset all test implementers need to have in order to build the best test suite.  

In terms of the test pyramid, we can certainly assume that there are responsibly written tests. What's illustrated there is how the test maintenance cost rises in parallel to the growing level of component interaction and complexity. The more complex the interactions in the test then the more likely it is to break. Broken tests need to be maintained. Tests of complex interactions take longer to repair.  

At each level of the pyramid, we have a set budget on how much we can spend on maintenance. Unit tests are cheap therefore we buy lots. Integration tests are a little more expensive so we can't afford as much. Functional tests carry the heaviest price tag so we just buy a handful.  

Tests at all layers are valuable and provide a distinctly different type of feedback. That's why it isn't a good idea to simply forego testing a layer at all. Having a codebase with only unit tests doesn't prove that the system actually runs.  

There are a few more factors in what defines the value of a test but I'm glossing over that by calling them "responsible" tests. If you had two integration tests that exercised the same grouping of components then you should probably entertain getting rid of one. We want clean, clear feedback and having multiple test breaks for a single issue will obscure the root of the problem. Given that we're trying to be as frugal as possible we'd probably identify tests like that early on.  

The one caveat/tangent I might mention is that the pyramid metaphor (and how I commonly see it described) doesn't account for things like capacity testing; the throughput of the application, the transactions/second they can service, their memory footprint and resource consumption and other issues that can potentially derail a release.  

Application deployment is an implicit test to make sure that you can actually install and run the application. There would never be an NUnit test for that but their is value in knowing that the application can be reliably deployed. We should try doing that earlier to get that feedback when the information is more fresh in our minds. 

...

And that was about it.  I had a few more words but they dealt with the internals of our specific situation.  Not worth repeating for obvious reasons.

On the fence about Java, open source is the shiznit and .NET build automation sucks

I moved to a new team more than half a year ago and at first things were rough.  The team itself was fine but I was giving up my precious C# for Java and I was entering a whole universe of new software.  My technology stack changed from Microsoft branded IDEs to that of the open source crowd.  I was out of my element but I thought, "Hey, time to try something new!".

Let's just get this out of the way; as a language, Java is not aging well.  Certainly not in comparison to C# which is getting updated more frequently with a lot better language niceties.  When .NET rev'ed itself up to v3 it brought along lambdas and extension methods which makes for some really nice, clean code.  This is noticeably absent from Java.  Even the simple act of passing around functions is no where to be seen in Java.  I have to create a full-blown class that implements one method.  It feels like Java is more worried about being pulled from the holy scripture of Design Patterns than it is about making a language that is enjoyable.  Ah, you can pass an anonymous class but it's still waaaay too verbose.

But that's the thing with Java.  It seems like it tries to be as horrifically verbose as possible.  One of the common Java practices is to mark everything final.  You see final everywhere.  Final member variables, final arguments, final variable declaration, final classes, final final final.  You'd wish that they marked things NON-final because there are far fewer instances of that.  Better yet, choose a language where variables are immutable by default.  You end up with a lot of chatty code, which is unfortunate since it's already a static language...you don't want to be known for pushing the envelope on verbosity.  Here's my favorite quote about it,

Whenever I write code in Java I feel like I'm filling out endless forms in triplicate. 

-- Joe Marshall

I can forgive these sins because I've finally been exposed to open source software.  Software built by the community.  For us, by us.  Open source software has a distinctly different flavor to Microsoft software.  Tools load faster.  There's less fluff.  It's more of what you need and less of what you don't need.  If something has a bug, it's fixed quickly and you can get the patch as soon as someone posts it.  You don't have to live with the same crap for years because it just doesn't show up as a priority on Microsoft's radar.  You know, like the "add reference dialog" that has been the bane of our existence since early 2000 that they only recently addressed (sort of).

I was introduced to Maven which is build and dependency management for Java.  For anyone who cares about build automation (and that had better be ALL of you), Maven beats the ever living snot out of anything you can find in .NET.  .NET's build automation story is absolutely abysmal in comparison.  I know some people think Maven is kinda crappy but it's still light years beyond what the .NET crowd has going on.  The only thing that seems to be standardized in .NET is using NAnt (and possibly but incredibly doubtfully, Albacore/Rake).  Here's a simple scenario that is more difficult to do in .NET,

1. Gather dependencies
2. Compile code
3. Run Unit tests
4. Run Integration tests
5. Deploy the application
6. Run smoke tests

Step 1 was always something that was custom to whatever shop you worked at.  This is where someone had to copy third party libraries somewhere your build process could access them (probably a lame network share).  This is a problem that has been solved for years with things like the Maven repository in Java and Gems repository in Ruby.  Maven just does this out of the box.  When you run a build, it will retrieve your dependencies prior to compilation.  There's a project NuGet for .NET that is supposed to alleviate this problem and I admit I haven't tried using it.  I can only hope it gets integrated into another project like Maven for the .NET community.

Steps 2-3 are straight forward (and usually) don't introduce too much pain.  Step 4 usually requires partially constructing the environment of your application which includes databases, web applications and other such "heavy" dependencies.  This is where the open source community trounces .NET.  When I tried automating testing of web applications it required that IIS be installed, that I had the correct users/authentication, that I figure out the mystery of the Web Deploy tools that are anything but intuitive and extremely prejudiced to IIS 6, blah blah blah.  It was a process that required massive manual intervention and implementation.  It's brittle and no company wants to pay for the time it would take to implement that.

With Maven, I just reference the Tomcat web server plugin, tell it to start a web application on the fly, install it, and run it.  It's disgustingly simple to do and it's actually running a very good analog of my production environment.  You don't get Tomcat Home Edition or Tomcat Developer Edition or some other lame neutered version of the web server product (I'm looking at you Cassini web server).  You get the real deal and you get it with only a handful of XML declarations.

Step 5 in .NET has no real public support or method.  It sucks.  Using the same Maven Tomcat plugin I mentioned above, deploying your web application is a breeze.

I don't think I care to carry on describing how immature the state of build automation is in .NET.  Suffice it to say, we .NET devs should stop waiting for Microsoft to make it for us.  It's a glacial development.  There's too much to be learned from the open source software projects.  First lesson, not all .NET projects need to be monetized.  Yes, you can be compensated for your hard work but there are tons of opportunities to do that outside of selling a product.  Embrace your community, have them push the feature list.  We can go from there.

So while I get a lot better developer tools in open source software, it doesn't come without a price.  I'm not impressed with documentation and it's difficult to find the answers you need.  Unless you subscribe to their IRC channel or mailing list, you aren't going to find help from available online content. Sure, it can get you going but once the rubber meets the road you will inevitably hit an issue that doesn't seem to be covered in anything you can find with conventional Google searches.  I've come to loathe all the websites that index mailing lists, jira and java documentation only to provide you with advertisement-laden dead ends.  You'll have to subscribe to a mailing list, post your question and pray that somebody wants to respond to a question they've probably seen a dozen times before.

The open source world isn't perfect but I love the tooling and the low barrier for entry.  If you've been living in the .NET bubble for some time, take a break and try something new.  The worst case is that you recognize ways to make your life easier.

It was a busy summer

This last summer I spent time getting embedded in a new group at work.  It's been a blast because the group is really enthusiastic about agile development and adopting new and efficient practices.  I've had a chance to practice a lot of the things I've been doing and studying in my free time including acceptance test driven development, NHibernate, ASP.NET MVC 2 and lots more BDD.

There's nothing sweeter than having a greenfield project and I was able to get my hands on one.  I was tasked with building an internal dashboard for some of our back end (Windows) services.  I'm still in .NET land and we're primarily a Microsoft shop so I used a pretty small but flexible stack,

• ASP.NET MVC 2
• Fluent NHibernate
• StructureMap
• jQuery

For testing, I tried out doing acceptance test driven development with SpecFlow which is the .NET analog to Ruby's Cucumber.  My unit/integration testing framework of choice is still MSpec.

I took this experience and made a presentation to the company about BDD.  For anyone that's looked around the web at BDD presentations you won't be incredibly surprised by the content.  Lots of the same points are made and I even borrowed the excellent high gear/low gear analogy of ATDD I found in someone else's presentation (I can't remember who :-(...sorry, uncredited person, whomever you are).  You can download it from my dropbox here.

I know I promised an example of doing ATDD and I'd really like to do it.  Now that work is semi-calm I can try to whip something up.  There are tons of other things I want to talk about like the Java codebase I'm working in now, when NOT to unit test, the current .NET craze around CQRS, and other interesting topics swimming around in my head.  

BDD: Just tell me what the eff it is!

Here's my big problem with BDD...on first blush the only thing you can see of it are the tools used in it for testing and a bunch of acronyms and buzzwords.  You get all these testing frameworks to try to write natural language tests geared to output something your stakeholder can both read and understand.  For a long time, the question still stood for me; what is BDD?  How do you do BDD?  Sure, my tests were now human readable but it still hadn't added up.

For those that don't understand what BDD is and even those that do, I want to walk you through what I've learned over the last year.  I want to give you an alternative narrative to the stuffy version on wikipedia and the dated top ranked Google results which offer breadcrumbs to understanding but didn't give me the clarity I was looking for.

Let's look at the two line summary Dan North gave last year,

BDD is a second-generation, outside-in, pull- based, multiple-stakeholder, multiple-scale, high-automation, agile methodology. It describes a cycle of interactions with well- defined outputs, resulting in the delivery of working, tested software that matters.

Get it?  Neither did I when I first read it.  After doing a lot of research, investigation and experimentation I finally get it.  Now that I do, that two line definition is an excellent summary.  There's a lot of information packed into those sentences.  We won't try to drill into all of that here.  Let's start with the simple building blocks of the BDD process.

It starts with a story

BDD is firmly rooted in the agile universe.  It relies on a lot of the mechanisms and practices you find in agile processes.  The first and most important is the practice of capturing user requirements as stories.  I'm not going to spend time in this post defining what a story is but let's just use an example we can refer to.

In order to build readership of the blog
As a blog reader
I want to leave comments

This is the first step in BDD, the first tangible piece of the process.  Your stakeholder(s) and your agile group develop stories.

Create examples/scenarios

You have your story.  Take that story and imagine (realistic) scenarios of it in practice.  Who it is that generates the scenarios may be multiple people.  Your scenarios are your acceptance criteria.  They tell you when the story is done.

Here are some samples of scenarios based on the above story.  (Remember, for this story, we're in the context of a reader of the blog and their desire to leave comments.)  The scenarios are written using the Given-When-Then format for describing scenarios.

Given a blog entry
When I write a comment
And I submit the comment
Then I should see my comment under the blog post

Given a blog entry
And the blog entry is no longer accepting comments
When I view the blog entry
Then I should not be allowed to write a comment

You can see how there would be numerous scenarios and contexts of users of your application.  When generating your scenarios, you must take them all into account.

You're just about to write code but...AUTOMATION!

Any good development environment should be as automated as (in)humanly as possible.  With a few quick commands/scripts you should be able to build your app, build your DB (if you have one), execute tests, deploy the app, and run it.  The more you have to do manually the harder this whole thing is.

Also, you should be able to run this in isolation on your workstation, if possible.  If your app is sharing a DB with 5 other workstations in your department, chances are you're going to have collisions.  You won't be able to make predictable tests and you'll be stepping on each others' toes.

When you write your code you want to be continuously running tests.  This should be a push-button activity that does not derail you for several minutes.  If it is then analyze what it is that you're doing and see if you can find ways to streamline it.  Either through tooling or through eliminating deficiencies in your testing strategy.

Outside-in: It's time to TDD

In BDD, we do things outside-in.  Meaning, you start at the level of the user's interaction with your application and then you build downwards.  It's a huge shift for a lot of people.  I haven't been doing it for very long but it's not a huge departure if you were already doing TDD.  Most of my difficulty was just learning how to automate UI interactions.

Here is where the rubber (finally) meets the road.  You have a story, you have some scenarios of your story that crystallize your acceptance criteria, and you have an automated development environment that allows you to build and test your app rapidly.

Using the story and scenarios from this blog post, we can write our first lines of code.  But...I'm not going to do that here.  I have a separate blog post (it certainly deserves its own) coming that will walk through the TDD process under the BDD umbrella.  I'll update this post to refer to it once it's done.

In summary

1. Write user stories
2. Create scenarios based on the stories
3. TDD those stories starting at the top level of your application (typically a UI) and keep developing downwards until it's functionally complete

Yes, that's seriously it.  There are methodologies, practices and tools that go inside of it but, at its simplest, that's all you're really doing.  If you're reading about BDD then I have no doubt that you're already aware of most of these topics anyways. 

The mystical cloak of BDD has now been removed!

DISCLAIMER:  This was meant to be a high level, entry level explanation of BDD because I have rarely seen it simplified.  Most information I've found online, the explanation seems to be a deep dive into how to write stories, stakeholder interaction, how DDD dictates the ubiquitous language, how it's the natural evolution of TDD (without ever showing you how!), and a variety of other entirely relevant information.  The problem is that there's a lot of cognitive overload.

In other words, for BDD enthusiasts, or developers in general because we like to be intellectual, don't dog me out and poke holes in my examples.  It's horribly contrived, I readily admit I am still learning these concepts and I don't believe anything I've written is incredibly misleading.

wtf, with a side order of bbq

Just a blurb; been trying out new technologies and shiny toys. Here's my naive/ignorant take on things:

Been years since Linq hit .NET. Why is the NHibernate criteria API so incredibly non-intuitive and not making a huge move to linq? It's still clinging to the uber-verbose criteria API that, even fluently, makes hardly any sense. Yeah, I tried using Linq2NHibernate too but it was buggy for anything outside of simple selects and joins. I had an inner join that would magically make itself an outer join when I added an orderby clause. Hot tip for NHibernate; ditch the criteria API and just use Linq like all the other cool kids. It's ok to abandon the bizarre RDBMS abstraction that's been created.

I've been spoiled by Ruby. I can't tell you how many times in my day I think, "goddam, why can't I have mixins!?" It would make a lot of things a lot easier/sexier.

Finally, how can there not be a fully automated solution for deploying .NET web apps? I want something that builds the DB, deploys an MVC app and creates one in IIS if it doesn't exist yet. I haven't found anything like that. Granted I haven't looked too much but I didn't find anything with a handful of google searches. This is another area where the other open source community blows .NET out of the water. If I cared, I'd try to make something like this but I don't have the friggin' time.

Finally, finally I am again reminded of how small the open source community is in .NET. Still can't figure out why.