-
Notifications
You must be signed in to change notification settings - Fork 284
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fix!: merge payload and data fields of Request #542
base: master
Are you sure you want to change the base?
Conversation
JS version have only |
Also, since this is breaking, maybe we should wait, and merge it once we have more breaking changes than a simple one like this (or some feature worth the minor bump). |
9d83dcb
to
49243db
Compare
I learned about |
It also says "Alternatively, you can send form data as URL parameters using the query_params argument." btw, not sure if that is something related or not. |
We will have the first version of fingerprints for Playwright soon 🤞.
Thanks. Of course, I forgot to update the docs. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice! Just a couple of questions...
@@ -26,6 +26,10 @@ | |||
|
|||
HttpMethod: TypeAlias = Literal['GET', 'HEAD', 'POST', 'PUT', 'DELETE', 'CONNECT', 'OPTIONS', 'TRACE', 'PATCH'] | |||
|
|||
HttpQueryParams: TypeAlias = dict[str, Any] | |||
|
|||
HttpPayload: TypeAlias = dict[str, Any] |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The dict
is kinda surprising here - I'd expect bytes
or something. How did this happen? 😄
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The downstream libraries (HTTPX & CurlImpersonate) do the conversions. And both of them accept RequestData = Mapping[str, Any]
type.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Right. I did some digging and this is what I found. payload
is part of the Request
schema on the API, and the API client requires it to be a string. It'll get serialized and passed to the API endpoint on platform. Of course, it also needs to be compatible with whatever the HTTP client accepts - that's where the actual processing gets done.
As a side note, I'm not sure what data
was supposed to be, but it seems it's not needed...
c04ecb2
to
414ac66
Compare
649eaa5
to
4c3d891
Compare
I've separated the Curl impersonation fix and other minor updates into a separate PR (#543), allowing us to release a patch version for 0.3. The |
### Description - Version 1.7.2 of `curl-cffi` introduces breaking changes. - This update includes adopting the new version, adding type aliases for `Request` fields, and incorporating other minor changes from PR #542. ### Issues - N/A (ad-hoc fix) ### Testing - The current set of unit tests should cover the changes. ### Checklist - [x] CI passed
4c3d891
to
f0ea79c
Compare
46e1e5e
to
4078303
Compare
5f7ac3d
to
62b332e
Compare
### Description - Unify the HTTP headers type across the project. - ~I used just the type alias plus custom validator in the `Request` model to reach the same result (lowercase & sorted).~ - ~Of course, now it is only in the `Request`, but I believe it is not a problem.~ - Take it out from #542. ### Issues - N/A ### Testing - N/A ### Checklist - [x] CI passed
62b332e
to
c371b29
Compare
Description
data
andpayload
fields on theRequest
model.payload
was not being provided to the HTTP clients, only thedata
field.In this PR, I'm merging them together, keeping only thedata
field (use the same naming asHTTPX
&CurlImpersonate
).payload
field (use the same naming as in JS Crawlee).I also defined type aliases for HTTP data and HTTP query parameters and used them across the project.Some struggle with Pydantic & serialization, but should be OK now...Issues
Testing
Checklist