[xmlsec] XMLDSIG and MS CryptoAPI problem...

[Csaba CSABAI]

Hello there!

Yesterday I tried to write a mail about the case "XMLDSIG/MSCryptoAPI",
as I can see it wasn't forwarded; but it isn't a problem, because I was
able to step ahead. I think I am almost there, the "xmldsig" generator
is almost done, based on the "MS CryptoAPI", however I am confronting
now a new error. When I do generate with my own "xmlsec engine
(mycryptoapi)" and the part of the "xmlsec/openssl" from the same
template one-one "xmldsig" output, momentarily all fields are well, but
the signature field. With the "OpenSSL (rsautl modul)" I remove from the
signatures the RSA encryption, the end of it is following:

"Xmlsec" with the original "openssl" engine (removed the RSA) is:
csibi at backup:~/test3$ openssl rsautl -in enc1-1 -inkey mika.cer -verify
-hexdump Enter pass phrase for mika.cer:
0000 - 30 21 30 09 06 05 2b 0e-03 02 1a 05 00 04 14 65
0!0...+........e
0010 - 7f aa d2 6d 16 67 d9 da-ed ed c4 58 18 bb 69 3e
...m.g.....X..i>
0020 - b2 0b 91                                          ...

The "XMLSEC" with the own "MSCryptoAPI" engine (case of removed RSA) is:
csibi at backup:~/test3$ openssl rsautl -in enc2-2 -inkey mika.cer -verify
-hexdump Enter pass phrase for mika.cer:
0000 - 30 21 30 09 06 05 2b 0e-03 02 1a 05 00 04 14 ef
0!0...+.........
0010 - f5 d3 b0 76 81 48 e9 c9-fe 35 c3 e6 fd 33 2f a5
...v.H...5...3/.
0020 - 4a 62 c9                                          Jb.

Very well visible it is, that in both cases at the beginning of the
signatures is perfectly the: SHA1 OID (30 21 30 09 06 05 2b 0e-03 02 1a
05 00 04 14)

Following the generated field of the XML DigestValue looks like:
<DigestValue>7/XTsHaBSOnJ/jXD5v0zL6VKYsk=</DigestValue>

Retract "base64 encoding", the following byte series of the signature
text's hash is: ef f5 d3 b0 76 81 48 e9 c9-fe 35 c3 e6 fd 33 2f a5 4a 62
c9

That is genuinely visible; it is identical with the HASH from the
retracted signature of the MSCryptoAPI, not the OpenSSL one. From this I
can conclude the following, in the "XMLDSIG" the <Signature> does not
contains from the <DigestValue> signature from the digest, but something
else. (Or may be in a different, converted form, for ex. a keyed hash or
something else)

Can you give me some info about:
- is it correct, what I see, the digitally signed hash is not the same,
like what is visible in the <DigestValue> filed?
- what have to be transformed more on the "digestvalue" before the
signature, or what else do I have to do to sign the correct "digest"?

csibi