WebScraper
refactor into scrapeURL
#714
Draft
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Directives:
engines/fire-engine/scrape.ts
, it only haslogger
, not the wholemeta
object)transformers
,engines
){success: false, error: ...}
objectsstack
) intact. if doing stuff like retries with a limit, previous errors are passed along using thecause
property.EngineResultsTracker
), non-expected errors should be explicitly logged andSentry.captureException
'dscrapeURL
, in order to avoid breaking other parts of the codebase. errors are passed in theerror
metadatamessage
-- if you need a specific error that is determinable by parts of the codebase, create a custom error class and useinstanceof
-- seeerror.ts
for referencescrapeURL
should never (even attempt to!) interface with the database. It should be its own standalone thing that could even be lifted out offirecrawl
as a whole. To keep it fast, reliable, and maintainable, we need to keep its footprint minimal -- DB code can be handled by the surrounding bits that are tangled up in that anyways (e.g.queue-worker.ts
)