[xmlsec] PKCS11 - Key not found

[Aleksey Sanin]

The issue is that checking RSA_FLAG_CACHE_PRIVATE was a hack.
This flag doesn't really say that this is a private key but
rather that caching should be used in private key operations.
It worked in your case and didn't work for someone else.

As I suggested in another reply, I think you should mark the
key as private yourself. This is a better way to do it.

Aleksey

On 9/17/17 10:36 AM, Pablo Gabriel Gallardo wrote:
> 2017-09-17 13:46 GMT-03:00 Pablo Gabriel Gallardo <pggllrd at gmail.com>:
>> I'll investigate to check what else can we do to determine whether an
>> EVP_PKEY is private or not. I'm not an OpenSSL expert but I want to help
>> with that.
>>
>> Regards,
>>
>> Pablo G. Gallardo
> 
> Aleksey,
> 
> I have a question. This is the code:
> 
> ```c
> RSA_get0_key(rsa, &n, &e, &d);
> if(n != NULL && e != NULL) {
>   if(d != NULL) {
>     return(xmlSecKeyDataTypePrivate | xmlSecKeyDataTypePublic);
>   } else if(RSA_test_flags(rsa, (RSA_FLAG_EXT_PKEY |
> RSA_FLAG_CACHE_PRIVATE)) != 0) {
>     /*
>     * !!! HACK !!! Also see DSA key
>     * We assume here that engine *always* has private key.
>     * This might be incorrect but it seems that there is no
>     * way to ask engine if given key is private or not.
>     */
>     return(xmlSecKeyDataTypePrivate | xmlSecKeyDataTypePublic);
>   } else {
>     return(xmlSecKeyDataTypePublic);
>   }
> }
> ```
> 
> First we check whether d is NULL or not `if(d != NULL)`. If we are
> dealing with a public key generally d is, indeed, NULL. In the case of
> smartkeys, even if we are dealing with a private key d is also NULL
> because d is inside the smartkey (never transmitted to the memory or
> CPU).
> 
> So we are failing in the second condition `RSA_test_flags(rsa,
> (RSA_FLAG_EXT_PKEY | RSA_FLAG_CACHE_PRIVATE)) != 0`, the question is:
> Those users that are reporting problems, What type of key they are
> using? If they are using a private key in a file, how can d be NULL?
> And if they are using a private key in another device, how they were
> doing that before the change in the condition so I can do the same?
> 
> Best,
> 
> Pablo G. Gallardo
>