Never build your fireplace under a rock overhang or in a cave. The heat could cause the rock to expand, with sections cracking and breaking, and possibly falling down on you and your fire.
Never build your fireplace close to dried brush, grasslands or other flammable areas.
Any rocks used should be from a high, dry location. Water logged rocks from low areas or waterways can explode dangerously when heated by the fire.