[xmlsec] strange error verifying cert

[Moultrie, Ferrell (ISSAtlanta)]

Hi:
  I'm getting the following OpenSSL error from deep down in certificate
verification (call stack is below).
error:0D07908D:asn1 encoding routines:ASN1_verify:unknown message digest
algorithm
  It works correctly if I use xmlsec.exe to verify the xml file, i.e.,
xmlsec verify --allowed x509 --trusted new_export.pem testfile.xml
  But it fails with my application making what I intended to be
essentially the same calls on the same data. 
  If I omit the import of the *.pem trusted cert file, then both xmlsec
and my application fail with the expected "cert verification failed".
Adding the --trusted <file> option to xmlsec lets it verify the cert and
the XML. Adding a call to xmlSecSimpleKeysMngrLoadPemCert() to my
application however results in the ASN1 error. The PEM file being loaded
and the xml file are the same in all cases. 
  Any ideas? I know this is a long shot but I'm just hoping that there's
some reasonably simple silly error that results in this ASN1 error that
you can tell me about!
Thanks!
  Ferrell

ASN1_verify(int (void)* 0x004ac8a0 i2d_X509_CINF(x509_cinf_st *,
unsigned char * *), X509_algor_st * 0x019fbf88, asn1_string_st *
0x019fbfc0, char * 0x019fbb98, evp_pkey_st * 0x019fd348) line 86
X509_verify(x509_st * 0x019fa150, evp_pkey_st * 0x019fd348) line 71 + 34
bytes
internal_verify(x509_store_ctx_st * 0x0012e93c) line 493 + 13 bytes
X509_verify_cert(x509_store_ctx_st * 0x0012e93c) line 306 + 9 bytes
xmlSecX509StoreVerify(_xmlSecX509Store * 0x01f03b28, _xmlSecX509Data *
0x01f036b8) line 987 + 9 bytes
xmlSecSimpleKeysMngrX509Verify(_xmlSecKeysMngr * 0x01f03b98, void *
0x00000000, _xmlSecX509Data * 0x01f036b8) line 622 + 16 bytes
xmlSecX509DataNodeRead(_xmlNode * 0x0036ee48, _xmlSecKeyInfoNodeStatus *
0x0012ea20) line 1190 + 27 bytes
xmlSecKeyInfoNodesListRead(_xmlNode * 0x0036ee48,
_xmlSecKeyInfoNodeStatus * 0x0012ea20) line 528 + 13 bytes
xmlSecKeyInfoNodeRead(_xmlNode * 0x0036eda0, _xmlSecKeysMngr *
0x01f03b98, void * 0x00000000, const _xmlSecKeyIdStruct * 0x005239a8
_xmlSecRsaKeyId, int 0x00000000, int 0x00000002) line 440 + 13 bytes
xmlSecKeysMngrGetKey(_xmlNode * 0x0036eda0, _xmlSecKeysMngr *
0x01f03b98, void * 0x00000000, const _xmlSecKeyIdStruct * 0x005239a8
_xmlSecRsaKeyId, int 0x00000000, int 0x00000002) line 442 + 29 bytes
xmlSecSignedInfoRead(_xmlNode * 0x00369800, int 0x00000000, _xmlNode *
0x0036ebe0, _xmlNode * 0x0036eda0, _xmlSecDSigResult * 0x01f03a40) line
1382 + 81 bytes
xmlSecSignatureRead(_xmlNode * 0x00369718, int 0x00000000,
_xmlSecDSigResult * 0x01f03a40) line 1122 + 25 bytes
xmlSecDSigValidate(_xmlSecDSigCtx * 0x00367368, void * 0x00000000,
_xmlSecKey * 0x00000000, _xmlNode * 0x00369718, _xmlSecDSigResult * *
0x0012ebe0) line 727 + 15 bytes

=====================================
Ferrell Moultrie (ferrell at iss.net)
Software Engineer

Internet Security Systems, Inc.
6303 Barfield Road
Atlanta, Georgia 30328
Phone:  404-236-2600
Direct: 404-236-2849
Fax:    404-236-2632
http://www.iss.net

Internet Security Systems -- The Power to Protect
=====================================