Memory leak within a loop setting/getting numeric vectors/arrays


Reported by Rob B.
public class TheModel 
  public double[] TheInput;
  public double[] TheOutput;
  public void Run()
      // code
for (i in 1:1000) {
   inputVec <- runif(1000)
   clrSet(model, "TheInput", inputVec)
   clrCall(model, 'Run')
Suspect that clrSet triggers a runaway memory usage. May be some reference counting in the COM gu to deal with.
Will be somewhat tricky to capture in unit tests. Must, however, in the end.
Closed Sep 20, 2013 at 11:37 PM by jperraud
There is no unexpected change in the memory footprint now. Manual testing captured in \testing\TestMemoryFootprint.r to be considered for unit tests

200 (3d5db1fa0e87) Fix to close issue https://r2clr.codeplex.com/workitem/70


jperraud wrote Aug 27, 2013 at 2:27 AM

Got a repro on clrSet indeed.
Will have to deep dive again in the confusing world of COM.
a <- clrNew('Rclr.TestArrayMemoryHandling')
# for (i in 1:100) {clrCall(a, 'CreateArray_double', 1e7L)}

vecInput = runif(1e7L)
clrSet(a, 'FieldArray_double', vecInput)
# for (i in 1:100) {clrSet(a, 'FieldArray_double', vecInput)}

jperraud wrote Aug 28, 2013 at 12:01 PM

OK, I think I am on the right track. To start with it seemed that adding a SafeArrayDestroy was not having any effect. Probably had none. On a hunch from a stack overflow page (which I cannot locate anymore), I removed the use of wrappers variant_t around VARIANT objects. Not there yet, but the mem() functions already shows a drastically reduced footprint.
if ONLY I had access to the P/INVOKE source code...

mem <- function () {clrCallStatic('Rclr.TestCases', 'GetPrivateMemoryMegabtyes')}
sinkobj <- function(a) {clrCallStatic('Rclr.TestCases', 'SinkLargeObject', a)}

a <- runif(5e7)

jperraud wrote Aug 29, 2013 at 10:42 PM

Replacing variant_t with VARIANT improved things a lot, but there are places where I have yet to figure how to do without it. The additional memory footprint seems to be 'only' about the size of the data now (yes, it was more...). Curious: a variant_t (or _variant_t, same thing to add to COM confusion) is a VARIANT, so the call to VariantClear should dispose of data. if only the code for SafeArrayDestroy was available, too...

jperraud wrote Sep 1, 2013 at 3:23 AM

Finally zooming in on the issues. When creating safearrays, safearrayputelement does not only put a "pointer" to the original data but copies all from e.g. a VT_ARRAY|VT_R8.
>   rClrMs.dll!rclr_ms_create_common_call_parameters(tagVARIANT * obj_or_class_name, char * mnam, tagVARIANT * * params, int paramsArgLength)  Line 2085    C++
    rClrMs.dll!rclr_ms_create_static_method_call_parameters(char * assemblyQualifiedTypeName, char * mnam, tagVARIANT * * params, int paramsArgLength)  Line 2099 + 0x19 bytes  C++
    rClrMs.dll!rclr_ms_call_static_method_tname(char * ns_qualified_typename, char * mnam, tagVARIANT * * params, int argLength, tagVARIANT * result)  Line 1730 + 0x2e bytes   C++
    rClrMs.dll!r_call_static_method(SEXPREC * p)  Line 568 + 0x26 bytes C++
As a workaround I am accepting that data is memory copied, but making sure that intermediary variants are cleared. This seems to flush out most if not all problems.
Need to find a way to pass some things by refernce, but as so many things with VARIANTS the recipe is missing and the effect on the .NET hosting API is not clear. This is work for another issue.