Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Speed up log file processing #918

Merged
merged 1 commit into from
Oct 12, 2024

Conversation

argilo
Copy link
Contributor

@argilo argilo commented Oct 12, 2024

After thousands of log files have accumulated, opening the Historical page becomes slow. A lot of the wait time comes from the list_log_files function, which scans the log files and summarizes them. I profiled this function and found that nearly half of the execution time is spent in dateutil.parser.parse, which is used to parse the time from the beginning of the log filename. I presume the parse function is slow because it must guess what time format has been used.

I tried switching to datetime.datetime.strptime, and this gave a significant speedup, but still a decent amount of time was spent in strptime, perhaps because it needs to parse the format string argument. Manually converting the pieces to integers and passing them directly into datetime.datetime() reduced the processing time to a negligible amount, and reduced the execution time of list_log_files by about 45%.

@darksidelemm darksidelemm merged commit cd7028b into projecthorus:testing Oct 12, 2024
6 checks passed
@argilo argilo deleted the historical-speedup branch October 12, 2024 14:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants