I am not a "license Nazi" but I do like to make it clear to people that CB and FRS (the frequencies the handheld walky talkies you buy at Walmart and other places use) are your only legal options without getting a license of some kind. There are way too many people in the offroad scene using illegal radios and encouraging others to as well without making it clear that doing so is illegal. Sure you probably won't get caught, but don't be "that guy".
CBs are allowed to use 4 watts and FRS is limited to 2 watts. The higher frequencies used by FRS "bend" around the terrain much better than CB frequencies do, so despite the power disadvantage, they usually work just as well if not better than CBs for small groups on the trail. FRS has better sound quality too (FM vs. AM for CBs), so transmissions tend to be much more intelligible. The biggest reason why people think CB "sucks" for the trail is improper antenna mounting. Your antenna, for any radio system, needs a proper ground plane to receive and transmit well. This means you should be mounting your antenna on the center of your roof, but you need a metal roof. The problem is finding a good ground plane on Jeeps and other vehicles that do not have metal roofs. It will take some experimentation to determine where your antenna transmits and receives the best. Frankly, CB and FRS really are sufficient for the vast majority of trail riders, but people lazy and sloppy and like to use poorly setup cheap radios. They compensate by using "race radios" that transmit with way more power and are completely illegal.
GMRS isn't a bad option, but requires a $70 license good for 10 years to be compliant (The fee is supposed to drop to $35 this year). The good news is that GMRS doesn't require a test like HAM though. GMRS uses high frequencies that work well offroad, and can transmit with up to 50 watts which is a huge upgrade from the 2 watt FRS limit (several channels are shared between FRS and GMRS, but broadcasting above 2 watts is classified as GMRS).
HAM requires a license with a test, but there are 3 different license classes; Technician, General, and Extra. The higher classes allow access to more bands and power, but also require more difficult tests. The test is $15 and a license lasts 10 years (although the rate is supposed to go up to $35 when the GMRS rate decreases to $35). HAM radios can transmit on a much larger range of frequencies (including the VHF frequencies that work well offroad) than GMRS, FRS or CB. Technician licenses are limited to 200 watts, while higher classes can go up to 1500 watts. HAM is by far the most superior radio type for offroad usage, but the problem is getting the people you wheel with to take the test and get licensed too. Also, few people you might wheel with outside your usual group will happen to have a HAM. Illegal users are much more likely to be caught on HAM since some of the HAM dorks make it a hobby to track down illegal operators for fun.
"Race radios" transmit on the PLMRS bands for the most part which require a license for a specific frequency in a specific geographic area (75mi radius I believe). These bands are intended for commercial use and the licenses are much more expensive and less useful for off roaders since you are constrained to a single frequency and area per license. These bands don't really have any advantage over a GMRS or HAM radio.