Eric Rauchway at The Edge of the American West has some good advice about how to respond to a journal's revise and resubmit letter. Hat tip. Probably the most important advice is to actually revise and resubmit the piece. The first time I received an R&R letter, I misinterpreted it, as did someone in the Eric's comments, and assumed that the journal wouldn't accept my article. If I had not run into an editor at a conference who urged me to resubmit, since they were hoping to publish, I would have sent it elsewhere.
Editors send out the other kind of letter all the time, and if they really intend to signal that there's no chance the article will appear in the journal, they know how to reject your piece. An R&R is no guarantee, of course, but it is usually a good idea to revise the piece and try again.