URL Should be same protocol as request

[MeshariSA0]

• In self-host we are running without https, however the response Next is always https.

Error Fixed:
HTTPSConnectionPool(host='172.18.0.8', port=3002): Max retries exceeded with url: /v1/crawl/6ebabc4e-6868-43da-a361-d1560dd55ce6?skip=125 (Caused by SSLError(SSLError(1, '[SSL: WRONG_VERSION_NUMBER] wrong version number (_ssl.c:1006)')))

[nickscamara]

Thanks @MeshariSA0! Will try to get this in soon!

@mogery i believe we had an issue with req.protocol a while ago in prod, where req.protocol would always return http, hence the enforcement. We might wanna double check on why.

[MeshariSA0]

Thanks @MeshariSA0! Will try to get this in soon!

@mogery i believe we had an issue with req.protocol a while ago in prod, where req.protocol would always return http, hence the enforcement. We might wanna double check on why.

This could be due to internal services communicating in HTTP, so the firecrawl-api container receives the request in HTTP protocol, so it always returns HTTP.

[busaud]

Thank you @MeshariSA0 for the solution, we have implemented it in our local deployment until this gets merged.