The attack is known to work against the widely used open-source encryption software GNU Privacy Guard, but requires that the would-be spy first intercept the message and then convince the sender to decrypt what seems to be a second message.
A noted cryptographer, however, stressed that PGP is not broken.
"If I use this, I get one message--I don't get your (secret) key," said Bruce Schneier, founder and chief technology officer of network protection provider Counterpane Internet Security. Schneier proved the existence of the flaw with Jonathan Katz, a professor at the University of Maryland at College Park, and with one of Katz's graduate students.
Details of the attack method will be given at a lecture at the Information Security Conference in Sao Paulo, Brazil, later this year, and paper on the attack method is available now at the Counterpane site.
The attack takes advantage of a flaw that existed in the PGP standard until last year. Because the defense against the attack requires that developers break compatibility with older versions, the makers of many encryption programs haven't fixed the problem.
That's the case with GnuPG, said Jon Callas, principal author of the OpenPGP formats standard for the Internet Engineering Task Force, the group responsible for setting technical standards on the Internet.
"Schneier and Katz have come up with a practical attack against this weakness that we have known about for a while," he said. "It's mainly a con attack--one person has to convince another to do something."
PGP is an example of a public-key encryption system. Each person using PGP has a private key, which they keep secret, and a public key, which they publish. A message encrypted with the public key can be decrypted by the private key, and vice versa.
The attack can be described using the typical cast of encryption problems: a sender (Alice), a receiver (Bob) and an eavesdropper (Eve). When Alice wants to send Bob a message, she encrypts the plaintext of her message with Bob's public key. No one can decrypt the message except for Bob, but Eve does manage to intercept the message.
Deciding that she wants to figure out what the message says, Eve applies a specific set of mathematical functions to the so-called ciphertext, corrupting it. Eve then sends the corrupted message, essentially a damaged version of Alice's encrypted message, to Bob without encrypting it. Bob decrypts it with his public key and gets a lot of garbage. Puzzled, Bob contacts Eve, who asks Bob to send the garbage text back. Eve then reverses the mathematical functions and removes the corruption from the message, and is left with the original message that Alice sent.
The mathematical sleight of hand is possible because there is a specific class of mathematical function that can be applied to an encrypted message and can be removed after the message is decrypted. Known as a homomorphism, the flaw opens the door to social-engineering attacks--that is, those that trick humans rather than break a code directly.
"The moral is not to send gibberish back to the person you got it from," said Schneier. "You decrypted it and sent it back to me. Unbeknownst to you, you have decrypted the message, but because of the corruption, you don't know it."
In March, security company Network Associatesfor the PGP software after it failed to find a buyer for its PGP business unit. Network Associates still owns the intellectual property surrounding the encryption, which it bought from PGP's creator, .