Storage: If you have an existing storage card that you'd like to use with your new camera, make sure that it's compatible with your new purchase. Most cameras on the market today use SD (Secure Digital) or SDHC (Secure Digital High Capacity) format cards. SDHC (Secure Digital High Capacity) cards are more expensive, offering storage capacities up to 32GB, but they're not backward-compatible with standard SD slots. There's also a new format on the block: SDXC, which supports storage capacities up to a whopping 2TB; those are even more expensive, and they aren't compatible with all SD/SDHC card slots.
In addition to storage capacity, there's also the speed issue to consider. SD and SDHC cards have a "Decoding Class" rating listed, which refers to the data-writing rate for each card. The higher the Class number, the faster the write speed; if you're planning on shooting video or using a high-speed burst mode, look for a Class 4 or Class 6 card at the very least.
To complicate matters further, there are a couple of other formats out there. Some cameras support MicroSD or MicroSDHC cards, a smaller version of the SD card format that isn't compatible with full-size SD slots. Older Sony cameras take MemoryStick cards, and older Olympus cameras use the XD card format; both companies' new cameras now support SD/SDHC cards. What's more, many higher-end DSLRs have a larger-format CompactFlash card slot. You'll want to consider all of these options when purchasing storage for your camera, though it is definitely easiest to go with standard SD/SDHC cards since you will be able to use them across cameras.
Battery life: Cameras use one or more of several types of batteries: AAs, either non-rechargeable alkaline ($5 for four) or rechargeable nickel metal hydride (NiMH, about $14 for four); high-capacity disposable CRV3s (around $10 apiece, and some cameras take two); or proprietary rechargeable batteries that can cost $25 to $65 to replace. Some digital cameras quickly drain batteries--especially alkaline batteries--which can be expensive and annoying. Battery life and cost often aren't related; some inexpensive cameras have great battery life, and some expensive ones use up a charge quickly. Either way, it's a very good idea to buy spare batteries.
Menus: When evaluating a camera, consider how easily you can reach common settings--resolution, macro mode, flash, and exposure adjustments--and how easily you can play back just-taken images. Too many buttons, and you waste time trying to figure out which button does what; too many menus, and you waste time digging through them.