Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Change ETLv2 file locking to use flock #167

Merged
merged 3 commits into from
Jun 22, 2017
Merged

Change ETLv2 file locking to use flock #167

merged 3 commits into from
Jun 22, 2017

Conversation

smgallo
Copy link
Contributor

@smgallo smgallo commented Jun 22, 2017

Change ETLv2 file locking to use flock

Description

The current ETLv2 file locking uses a pid in the lockfile to determine if a process is currently running and using the lockfile. This causes a problem for processes using moderate amounts of memory because PHP hits the script memory limit when attempting to fork/exec to run the ps command. This PR changes the behavior to use flock() instead.

Motivation and Context

PHP script out of memory errors.

Tests performed

Tested running an ETL pipeline

  • Single pipeline running at a time
  • Two of the same pipelines running at the same time
  • Kill a pipeline and observe proper lockfile cleanup
  • @ryanrath confirmed that memory errors are no longer present

Types of changes

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to change)

Checklist:

  • My code follows the code style of this project as found in the CONTRIBUTING document.
  • I have added tests to cover my changes.
  • All new and existing tests passed.

@smgallo smgallo added bug Bugfixes Category:ETL Extract Transform Load labels Jun 22, 2017
@smgallo smgallo added this to the v7.0.0 milestone Jun 22, 2017
@smgallo smgallo merged commit 9e87716 into ubccr:xdmod7.0 Jun 22, 2017
@smgallo smgallo deleted the etl/use-flock branch June 22, 2017 20:48
ryanrath
ryanrath reviewed Jun 23, 2017
View reviewed changes
Copy link
Contributor

@ryanrath ryanrath left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me.

ryanrath pushed a commit to ryanrath/xdmod that referenced this pull request Jun 23, 2017
@smgallo @ryanrath
Change ETLv2 file locking to use flock (ubccr#167)
ff362f5 
* Documentation and logging improvements
* Use flock to manage ETLv2 lockfiles
ryanrath pushed a commit to ryanrath/xdmod that referenced this pull request Jul 24, 2017
@smgallo @ryanrath
Change ETLv2 file locking to use flock (ubccr#167)
f7f3945 
* Documentation and logging improvements
* Use flock to manage ETLv2 lockfiles
chakrabortyr pushed a commit to chakrabortyr/xdmod that referenced this pull request Oct 17, 2017
@smgallo @chakrabortyr
Change ETLv2 file locking to use flock (ubccr#167)
f011345 
* Documentation and logging improvements
* Use flock to manage ETLv2 lockfiles
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@ryanrath ryanrath ryanrath left review comments

@jpwhite4 jpwhite4 jpwhite4 approved these changes

Assignees
No one assigned
Labels
bug Bugfixes Category:ETL Extract Transform Load
Projects
None yet
Milestone
v7.0.0
Development

Successfully merging this pull request may close these issues.
3 participants
@smgallo @jpwhite4 @ryanrath