[xmlsec] signing thousands of files - too many open files

[Aleksey Sanin]

I would start from editing /etc/security/limits.conf or
/etc/sysctl.conf to increase the limit :)

I need to think about this change. It looks safe but
somewhat breaks the abstraction level. I wonder if there
is a better way to do it.

Thanks!

Aleksey

On 3/2/15 3:36 AM, Dimitrios Siganos wrote:
> Hi,
> 
> I have a need to sign and verify thousands of files using xmldsig.
> Unfortunately, I can't archive them and sign the archive.
> 
> This is my setup:
> * 2000 files (file1, file2, ..., file2000)
> * a signature template that references all of the 2000 files (i.e. 2000
> file references)
> <Signature>
>   <SignedInfo>
>     <CanonicalizationMethod
> Algorithm="http://www.w3.org/2006/12/xml-c14n11"/>
>     <SignatureMethod
> Algorithm="http://www.w3.org/2001/04/xmldsig-more#rsa-sha256"/>
>     <Reference URI="file1">
>       <DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
>       <DigestValue/>
>     </Reference>
>     ... 1998 more references ...
>     <Reference URI="file2000">
>       <DigestMethod Algorithm="http://www.w3.org/2001/04/xmlenc#sha256"/>
>       <DigestValue/>
>     </Reference>
>   </SignedInfo>
>   <SignatureValue/>
>   <KeyInfo><X509Data><X509Certificate/></X509Data></KeyInfo>
> </Signature>
> 
> Running xmlsec1 --sign fails with the error "Too many open files". The
> reason is that libxmlsec opens all 2000 files, and leaves them open,
> until the end of the life of the entire operation.
> 
> I don't have the option to adjust the open files limit (ulimit).
> 
> I am able to fix/workaround the "too many open files" problem by closing
> the input files as soon as we are finished reading from them. This is
> the gist of my change:
> 
> diff --git a/src/transforms.c b/src/transforms.c
> index 8a2ded2..fa5b885 100644
> --- a/src/transforms.c
> +++ b/src/transforms.c
> @@ -1195,6 +1195,7 @@ xmlSecTransformCtxUriExecute(xmlSecTransformCtxPtr
> ctx, const xmlChar* uri) {
>          return(-1);
>      }
>  
> +    xmlSecTransformInputURIFinalize(uriTransform);
>      ctx->status = xmlSecTransformStatusFinished;
>      return(0);
>  }
> 
> Could you please comment on whether my change makes sense and is correct?
> 
> Regards,
> Dimitrios Siganos
> 
> 
> 
> _______________________________________________
> xmlsec mailing list
> xmlsec at aleksey.com
> http://www.aleksey.com/mailman/listinfo/xmlsec
>