Carrying your rifle when traveling on horseback can be an uncomfortable challenge if you don't use the proper technique. Keeping yourself relatively comfortable and your rifle protected are the two main keys.
Often, using a saddle scabbard is the ideal solution. Here's how. Narrow rifles (like lever guns) are the easiest, but it works with a bolt gun and a single shot. If carrying a bolt action, just make sure the bolt doesn't rub against the horse.