Also, when we have the point captured (CastleBlack controls it), and we enter the conquest point area again it will say 'CastleBlack is attacking conquest point', despite us being in control.
When that happens we stop recieving items/XP boost.
Town harvester is not related to conquest point, it has a chance to give you an item based on which biomes your town has chunks of.