[xmlsec] PKCS11 - Key not found

[Pablo Gabriel Gallardo]

2017-09-17 13:46 GMT-03:00 Pablo Gabriel Gallardo <pggllrd at gmail.com>:
> I'll investigate to check what else can we do to determine whether an
> EVP_PKEY is private or not. I'm not an OpenSSL expert but I want to help
> with that.
>
> Regards,
>
> Pablo G. Gallardo

Aleksey,

I have a question. This is the code:

```c
RSA_get0_key(rsa, &n, &e, &d);
if(n != NULL && e != NULL) {
  if(d != NULL) {
    return(xmlSecKeyDataTypePrivate | xmlSecKeyDataTypePublic);
  } else if(RSA_test_flags(rsa, (RSA_FLAG_EXT_PKEY |
RSA_FLAG_CACHE_PRIVATE)) != 0) {
    /*
    * !!! HACK !!! Also see DSA key
    * We assume here that engine *always* has private key.
    * This might be incorrect but it seems that there is no
    * way to ask engine if given key is private or not.
    */
    return(xmlSecKeyDataTypePrivate | xmlSecKeyDataTypePublic);
  } else {
    return(xmlSecKeyDataTypePublic);
  }
}
```

First we check whether d is NULL or not `if(d != NULL)`. If we are
dealing with a public key generally d is, indeed, NULL. In the case of
smartkeys, even if we are dealing with a private key d is also NULL
because d is inside the smartkey (never transmitted to the memory or
CPU).

So we are failing in the second condition `RSA_test_flags(rsa,
(RSA_FLAG_EXT_PKEY | RSA_FLAG_CACHE_PRIVATE)) != 0`, the question is:
Those users that are reporting problems, What type of key they are
using? If they are using a private key in a file, how can d be NULL?
And if they are using a private key in another device, how they were
doing that before the change in the condition so I can do the same?

Best,

Pablo G. Gallardo