Rare books are books which are, for one reason or another, difficult to obtain. Depending on the book, demand may be high, in which case the book can be quite expensive, or demand may be lower, restricted to a more specialized community. However, this may not necessarily mean that the price drops. Several firms around the world specialize in rare books, tracking down books for their clients and offering various finds at public sales and auctions, and rare book collectors can also be found all over the world.
There are many reasons why a book can become rare. While rare books are often old, this is not always the case, and being old doesn't necessarily make a book rare or valuable, with the exception of really old books. For instance, Renaissance books and manuscripts, by virtue of their age, tend to command a high price due to their historical value and rarity, while books from one to two centuries ago are not necessarily valuable. 19th century penny dreadfuls, for example, might be intriguing collectors items, but they are usually far from rare.
A book may be rare by its nature. First editions, for example, are often very rare because publishers typically keep initial print runs low in case the book doesn't sell. Book collectors who manage to find a first edition of a hugely popular book may be prepared to pay a pretty penny; first editions of Harry Potter and the Philosopher's Stone, for example, were selling for around $45,000 US in 2008. Books with printing errors, inclusions which were later dropped, and other interesting errata may also be rare, as are controversial books which somehow managed to escape pulping at the publisher's.
Books which are signed by the author, especially with an inscription, are also considered rare books, and likewise with books which belonged to someone famous. The bookplate of a prominent historical figure in the front of a book can elevate the value immensely, even if the book itself is not intrinsically valuable. Books which have been turned into films may be made more valuable if members of the cast sign editions of the books, illustrating a situation in which the book is made famous by association.
Collecting rare books can be very rewarding, but also potentially dangerous. Bibliophiles may spend alarming amounts of money on books every year, and they will stop at nothing to get a particularly coveted edition. They are also vulnerable to predatory sellers, such as people who fake autographs or make a book out to be more rare than it is. As a general rule, any reputable seller will happily agree to a third party inspection to verify the book and its provenance, and if you happen to be looking at a rare book signed by a modern author, some authors have verification programs for autographs.
You should be aware that genuinely old rare books rarely reach the public market for sale; most amateur rare book collectors focus on books from the last two centuries. Private collectors tend to hang on to their books, and when they die, their books are often donated to museums, rather than sold. In the rare instance that older manuscripts and books do make it to the open market, they are typically extremely expensive, and they are usually snatched up by academic institutions and private libraries.