Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Strategy for Meter & ActionScheduler when schedules cannot be served at requested intervals #118

Open
slogen opened this issue Dec 6, 2015 · 1 comment

Comments

@slogen
Copy link

slogen commented Dec 6, 2015

I am missing a strategy for what to do when ActionScheduler cannot meet the requested interval between invocations.

I am using the (very nice) Metrics.NET library to monitor thousands of Meters. Unfortunately that means that the Meters default Tick() refresh-rate at MeterMetric.TickInterval = 5s cannot be serviced.

As a result, the ThreadPool spins up more Threads, which actually makes the problem worse, by causing a cascade of Thread creations. Eventually upto ThreadPool.MaxThreads and consumes all OS resources changing between threads that update Meters :(

The below code demonstrates the issue (on my machine, milage may vary as it depends on slowness of Meter.Tick()-updates).

        [Fact]
        public void ActionScheduler_ShouldNotCascadeThreadCreationWhenManyMetersAreInstantiated()
        {
            var count = 50000;
            var meters = Enumerable.Range(0, count)
                .Select(i =>
                {
                    var meter = Metric.Meter(string.Format("M{0}", i), Unit.Calls, TimeUnit.Seconds);
                    meter.Mark();
                    return meter;
                })
                .ToList();
            // The mere *existence* of many meters makes the ActionScheduler schedule a large
            // amount of tasks that cannot execute at the desired (5s) intervals on some machines
            // 1. leading to the TaskPool extending the number of threads, 
            // 2. leading to even more scheduling time spent, and even worse execution (goto 1)
            Func<int> threadCount = () => System.Diagnostics.Process.GetCurrentProcess().Threads.Count;
            var threadCountInitial = threadCount();
            Thread.Sleep(TimeSpan.FromSeconds(20));
            var threadCountAfterSomeTime = threadCount();
            Assert.True(threadCountAfterSomeTime <= threadCountInitial + 1);
        }

I am using a workaround that extends the ActionScheduler with a TaskScheduler to run the task on, defaulting to TaskPool.Default (with exactly the old behaviour). But I can inject a thread-limited TaskScheduler via a static default to use for the Scheduler. This means that I can limit the amount of effort dedicated to Meter updates and live with the slow update-times that gives.

@slogen
Copy link
Author

slogen commented Aug 28, 2022

Since nobody else has this issue, and I stopped using Metrics.net, I am closing this issue as not-planned

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant