Feynman was brilliant. I don't doubt for a moment that he has had his own internal strife regarding his involvement in the project. Many of the scientists on the project were afflicted with a sort of tunnel vision as they focused on completing the job at hand, or felt that they had to get it done before the technology was used on them, only to later feel some version of guilt about their work. Feynman expresses his feelings about around 3:15 and goes on to say that he felt "that things were sort of doomed."

The only Manhattan Project scientist to leave the project was Józef Rotblat.

IN MARCH 1944 I experienced a disagreeable shock. At that time I was living with the Chadwicks in their house on the Mesa, before moving later to the "Big House;' the quarters for single scientists. General Leslie Groves, when visiting Los Alamos, frequently came to the Chadwicks for dinner and relaxed palaver. During one such conversation Groves said that, of course, the real purpose in making the bomb was to subdue the Soviets. [...] Remember, this was said at a time when thousands of Russians were dying every day on the Eastern Front, tying down the Germans and giving the Allies time to prepare for the landing on the continent of Europe. Until then I had thought that our work was to prevent a Nazi victory, and now I was told that the weapon we were preparing was intended for use against the people who were making extreme sacrifices for that very aim. [/reformation.org]

In the letter above, Rotblat goes on to explain that Neils Bohr shared concerns of a possible arms race between West and East, and his thoughts about why other scientists did not make the same choice to leave the project, and of accused of being a spy for the Russins after quitting the project. He was instructed speak to no one of his reasons for leaving.

Rotblat later went on to found the Pugwash Conference, with Bertrand Russell, which aimed to bring together scientists to work toward reducing the danger of (nuclear) armed conflict, and worked towards nuclear disarmament.

An excellent interview with him is over at the Vega Science Trust website:
http://vega.org.uk/video/programme/22

The National Film Board of Canada recently produced a documentary about on the work of Rotblat and the work of Pugwash. Trailer here:

I saw the full film tonight at a screening in my home town. I did not immediately realize that the Pugwash conferences actually *began in a small town just a couple hours away. (I was sure that it meant a different Pugwash.)

The full film is excellent and informative. The work of Rotblat and the conference vitally important to the outcome of many events of the Twentieth Century.