Memory Leaks: Part III – sessions and cfcs
In part III of my memory leak posts, I will attempt to show an example of how variable references can increase within in the heap on every request. If you would like to know more about the tools I used to identify the problem, or would like to follow along with the example you should take a look at my previous posts,
"ColdFusion Memory Leaks: Part I – profiler introduction"
"ColdFusion Memory Leaks: Part II – variables scope"
Note: The example that I will demonstrate is 90% complete. It is similar to the behavior that I experienced in my application, but not as severe. This only works in specific circumstances, where my problem appeared 100% of the time. Unfortunately, this is the closest I could come without including hundreds of thousands of lines of code.
For this example, we will expand the case that I demonstrated in the variables scope example. We will use the same test.cfc as the previous example.
Next we will need to expand the cfm page, we’ll make the following modifications.
- add session timeout of 10 seconds (to see results quickly)
- only load test.cfc into application on the first request
- create an array on the session
- create a new instance of test.cfc locally, and append it to the array on the session
creating a new session
completed request - session size = #arrayLen(session.sessionArray)#
reset application.test (this clears all memory references)
What should we expect in this example? (feel free to correct me if you think this should be different)
- 1 instance of test.cfc should be loaded into application
- foo exists in variables scope and creates a reference to an array of 100 instances of test.cfc (this is the example from Part II and should be destroyed at the end of the request)
- an instance of test.cfc will be added to the session on every request. (5 request in a session = 5 test.cfc)
- when the session expires, the array should be cleaned from the heap.
Now for a flash movie that will show the results of what actually happens (my apologies if this is rushed, there’s quite a bit going on in a short timeframe)
What happened??(also, how you can follow along if you execute the example)
- On the first request, test.cfc is loaded into the application at the same time a new session is created
- Every request loads an extra 100 references to test.cfc (which should only exist in variables scope)
- The session that loads the cfc into application scope (and only this session) will persist all the references for the life of the application
- All additional sessions will hold their references for the life of the session
- Adding a structure to session instead of a cfc will create the proper behavior
Summary
Why does this happen? I can’t tell you, I’m hoping the community, or adobe will be able to figure that out. Some other notes that I would like to add…. The behavior occurs if it is a struct OR an array on the session. The behavior also occurs if all the cfcs are different (I used test.cfc for all references to keep the example simple). If you reload application.test, the references will be cleaned up, but the behavior will occur for the session associated with the request that created application.test.
How do I fix this in my application???
I am not going to recommend a solution for this because I haven’t found a "magic bullet" yet. I am also not going to recommend a solution because every application is unique, and every memory leak is different. I will however tell you that my problem was very similar to this. We were keeping a history stack on the session that consisted of an array of simple beans. Converting the beans to structures, solved a large part of the problem. This isn’t always an option if you are storing complex objects on the session. I’m very interested to see if anyone else having a similar problem is storing cfcs on a session, as the root cause seems to be a wacky combination of instantiating cfcs on multiple scopes within a request.
Comments are closed.
One thing you might try to see if you get a different result is change cfset arrayAppend(arrayToReturn,createObject("component","test").init()) line to use a intermediate var scope variable. Something with arrayAppend might be causing the issue. It might use a non var scoped variable to store the result of the createObject before it is appended to the array.
Daniel-
That’s a great observation, and was one of the potential culprits that I looked at a while ago. ArrayAppend returns a result, but in this case we’re not assigning the result to anything because we don’t care. This could be assigning the result (boolean) to a variable in variables scope.
Unfortunately, after trying to assign this to a locally scoped variable, I end up with the same result. I ran it again a few different ways last night, and ended up with the same result every time. The original instances are not released.
I’m hoping to hear back from adobe soon, I filed a bug report, and they acknowledged that this was a known issue. They were already working on a hotfix as it had been reported by another customer. Hopefully we can get some resolution without having to wait for scorpio.
Hi Mike –
I’m very interested to know why Robi Sen isn’t able to recreate the issue, yet you were?
Did anything come out of diff’ing your jvm.config and jrun.xml files? (I’m guessing that JVM arg diffs could potentially be a very important diff… for instance, the webapper guys have some good suggestions that have helped us tremendously – http://www.webapper.net/index.cfm/2006/6/8/20060606021131)
Were you on different versions of Jrun (i.e. different Updater levels)?
Different O/S’s?
Diff number of processors?
Diff Web Servers (though I doubt it’s related)?
Diff CF 7 installation types (i.e. Stand-alone, Multi-server, or J2EE), or did you already mention that?
Aaron,
I tried doing Milke’s test on 3 different machines, 5 different OS ( mostly using VM ware), three major CF versions (6.1,7,7.0.2), different VMs, as well as instead of just using the micro testing that Mike was using I also used an application where I was using Var scope in the way he was suggesting caused issues. I also used load testing tool OpenSTA to run stress tests for several hours on these enviornments and was never able to create the issue. I have also worked with well over one hundred customers solving various memory or CF related issues and have never seen Var scope be an issue. I have also seen many customers who have extremely high volume large commerce applications that use Var scope not have this issue. In my opinion the issue is not with CF itself but something about Mike’s setup. I suspect the issue is something not obvious like how the application server is tuned and might very well be something rather bizarre.I think finally the real issue here is
1. Mike’s assertions are not reproducible consistently
2. CF customers don’t seem to be wildly reporting var issues
It would be nice to get a hold of Mike’s environment and really probe the issue but I think that Mike’s assertion that Var scoping variables causes memory leaks in CF is doubtful.
Robi
I’ve had quite a few responses to my page at
http://www.digitalmethod.co.uk/cf
all from people with the same problem.
I have 4 servers running ColdFusion, one of which has been running for months without a restart and one of which needs a restart every 12 hours.
The only difference is the amount of load they’re under. Are you sure in your stress tests that you have simultaneous threads running, and that the server is at its limits?
I’m almost certain that there’s a memory leak in CF pertaining to components, and I think it happens when simultaneous requests call the same method of a component in the application scope using a session scope variable as an argument, something like:
application.objUser.fnCheckLogin(session.sUser)
where objUser would be a component and session.sUser is a Struct in the session scope.
A good test would be to create a function that uses all the suspected functions – CreateUUID(), hash(), arrayAppend(), StructCopy(), etc and call it with a session struct as an argument.
Having said that this issue does seem to be only affecting Windows users so the problem may be completely unrelated!
Tom
Aaron-
I’m still not quite sure why Robi was not able to recreate the issue in his local environment. A few months ago I tried running with Robi’s suggested jvm.config and jrun.xml and was still able to reproduce the behavior. In fact, I have now been able to reproduce on almost every combination of the following environments. (and verify that code changes fix the behavior)
CFMX: 6.1, 7.01, 7.02
JVM: Sun 1.4, Sun 1.5, BEA 1.4, BEA 1.5
OS: Windows Server 2003, Windows XP
(I have also tested a completely clean CFMX 7.02 install on WinServer 2003, with nothing else on the machine)
I know that Robi still doesn’t think that there is an issue with code, and believes it is related to environment, but I see a flaw in the test logic. When testing, in order to verify a fix I would follow these rules.
1. Reproduce the behavior
2. Make a Change to one variable in the equation (code, environment, etc)
3. Does behavior still exist? If yes, revert change and try another
4. Continue testing one variable at a time until behavior no longer exists
In Robi’s case, he was NEVER able to complete step 1, which means his testing really doesn’t tell us anything at all. The problem could be related to environment OR code, but there is NO way to tell with his tests.
In my case, I was able to demonstrate the behavior, then ONLY make a change to code and verify that the behavior was fixed. There is still a potential for an environment workaround, but I wasn’t able to find one. Until someone can reproduce the problem, make a tweak to the environment, and verify that the problem no longer exists, I don’t think that anyone can claim it is strictly an environment issue.
One additional note I would like to clarify is that we are talking about "Variables" scope here, NOT Var scope which Robi mentioned above.
Tom-
For quite a while I had a hunch that simultaneous requests were the cause of the problem, but after running quite a few tests I was never able to prove anything. I still saw the behavior when a single thread was running on a machine.
We’ve also seen similar cases where one server was restarting every 12 hours and others stayed up for weeks, but ended up determining that the server without load was just taking longer to materialize the problem. In addition, the application timeout was set to 48 hours and sometimes we would have a few days of inactivity and the server would reclaim memory due to application timeouts.
I still don’t know the root cause of everything, but have now found 2 similar, but separate leaks in one of our applications. Before finding these, we had servers with out of memory exceptions every 12-24 hours. After fixing the first one, this time frame increased to 2-3 days, now after fixing the second we have servers staying up for 30+ days under load (we generally bounce by then for patch releases)
In both cases the root cause was an array of objects (cfcs) stored on the session. Objects were appended and deleted from the arrays during a session. Refactoring one to an array of structs and one to a struct of structs eliminated the unexpected memory consumption.
Mike,
We think actually we might just have found the leak.
Thanks to your tips we got JRockit running and ran a test against our app. There were thousands of instances of coldfusion.sql.* classes.
I immediately thought that the query of queries we use may be the issue so we wrote a little app that did the following:
1/ Create a component in the application scope with some methods to do query of queries on a structure of queries passed into it.
2/ Create an Structure with several keys, all different queries, and persist it in the session scope.
3/ Pass the structure as an argument to the methods of the component.
4/ Load test the app.
CF leaks like a rusty bucket.
Rewriting our app as we speak. Many thanks for all your tips.
Tom
Tom,
Persisting large objects, i.e. queries nested in a structure does not necessarily cause a memory leak. It just takes up large amounts of memory for each active user of your system. I limit the use of session for this very reason. I once witnessed an application that cached much of the database in session (YIKES, poor design). With every new session I would see 4 meg peel off the server. Not a memory leak just poor design. Hope this helps.
Hi Mark,
I have experience this memory leak and have also replicated your tests. I’ve written up a long post on it here.
http://www.jmpj.net/jason/index.cfm/2007/4/14/Cold-Fusion-Memory-Leak
There is a hotfix now available for this issue. Please see
http://www.adobe.com/cfusion/knowledgebase/index.cfm?id=kb401239
This should fix the problem.
Thanks,
Hemant
I tried your testing code from part 1 to 3. I am 200% sure there is memory leakage.
The solution I found is to store all complex objects in server scope rather in Application scope or sesssion scope.
@Hemant did you test mentioned scenarios.
my suggestion is to store all complex object in server scope and you will see there is no memory leakage problem.
I am happy to help anybody who is struggling like these issues.
Thanks
Sana
Testing methods.
In your examples you use Application scope for your CFC. a CFC in application scope will always be maintained so by adding 100 instances of itself to an ongoing array will always increase. Unless I am missing something it seems to me that it works exactly as you coded. 100 instances are created per request and stored in the array in the cfc. this adds exactly as you show it in the #instances.
JT-
Keep in mind that the 100 instances of the cfc are created within the VAR scope of the method that is being called. These variables are scoped locally to the cffunction, and not part of the component’s scope. Any variables within the var scope should only live for the lifetime of the method call.
So Mike (and other readers commenting on this entry), did things ever get resolved with the memory leaks you were observing? Some of the comments point to what was ultimately found to be caused by file uploads, fixed in an available hotfix in 7.02 (must be applied manually, not implemented by Cumulative Hot Fixes). Some identify that bad programming practices can cause what seem to be leaks.
But Mike you’ve contended all along that this was not the case for you. If perhaps you ended up writing another entry about all this, can you point to it here, so readers can find it? Or if you never solved it, but perhaps it went away when you moved to CF8 (or something else changed), please do tell. :-) I’m sure many readers (for years) will look forward to the understanding shared here.