The Unreal engines are good, but they're not the easiest to begin with. Source is apparently much easier, but it's a terrible engine in itself. In my opinion, if you want an easy-to-mod game with a good engine, do Tribes, or Neverwinter Nights (Aurora engine) if you want an RPG to start with. That's the short of it.
The long of it is that the Unreal 3 engine doesn't really present more in terms of "capability" as far as I know when compared to the Unreal 2.5 engine. It simply makes everything prettier. I have not seen anything that the Unreal 3 engine does that the Unreal 2.5 engine can't yet, except that silly PhysX compatibility, but that line of cards flopped. Note that I don't care about number of particles or particle improvements, because that goes in the "prettiness" section.
The Source engine is bad because it's a layered "sham engine" built upon Quake. That might be a tad bit of an exaggeration, but it's quite a bit true, and it explains a lot. The Source engine was flopping its "oh look physics" **** around when physics was already par for the course (Halo 2, for instance, had better physics by many standards). Not to mention that its grasp of physics is slippery, at best. The engine's answer to any two objects colliding for an extended period of time seems to be "**** it, let them merge". When the force behind them disappears, they both launch in the opposite directions at incredible speeds. This is just one example. Counter-Strike: Source's "surfing" is another, if you know what that is. Source has trouble handling anything more complicated than guns, that is, pointing and shooting and then hitscan makes damage. Melee combat is just about out of the question. The Pirates, Vikings and Knights team has somehow managed to code around the engine's shoddiness, producing a work of art. How they do it is a mystery to me, and apparently all of the other Source modding teams, because a mod with melee performance that good simply does not exist anywhere else within the Half-Life 2 community.
Tribes has a good engine because it handles physics well for its age. "Skiing", unlike CS:S's "surfing", is not a gross anomaly, but rather a critical feature of the game that shows off the game's good handling of acceleration and friction. More importantly, the engine can handle maps of incredible area and practically unlimited height, and as long as all of the clients have all of the models needed to display the represented props and buildings, the buildings and props (not to mention the weapons) can be combined, rotated, and moved to any location and to produce any shape without any need for extra content. Through this, many engaging custom maps and mods can be created without needing a single extra file. Neverwinter Nights is similar in this aspect, and that is why it has such a thriving community whereas NWN2's multiplayer community has barely taken off.
