Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Fix IP URL previews on Python 3 #4215

Merged
merged 31 commits into from
Dec 21, 2018
Merged

Fix IP URL previews on Python 3 #4215

merged 31 commits into from
Dec 21, 2018

Conversation

hawkowl
Copy link
Contributor

@hawkowl hawkowl commented Nov 21, 2018

No description provided.

@hawkowl
Copy link
Contributor Author

hawkowl commented Nov 21, 2018

Initial fix for #4208 . Needs more tests, mostly around the moved code regarding IP blacklisting.

@codecov-io
Copy link

codecov-io commented Nov 26, 2018

Codecov Report

Merging #4215 into develop will increase coverage by 0.04%.
The diff coverage is 93.61%.

Impacted file tree graph

@@             Coverage Diff             @@
##           develop    #4215      +/-   ##
===========================================
+ Coverage    73.54%   73.59%   +0.04%     
===========================================
  Files          302      302              
  Lines        29894    29924      +30     
  Branches      4891     4898       +7     
===========================================
+ Hits         21987    22023      +36     
+ Misses        6470     6463       -7     
- Partials      1437     1438       +1
Impacted Files Coverage Δ
synapse/http/endpoint.py 69.07% <ø> (-1.86%) ⬇️
synapse/rest/media/v1/preview_url_resource.py 63.69% <75%> (-0.09%) ⬇️
synapse/http/client.py 78.86% <94.44%> (+6.63%) ⬆️
synapse/util/file_consumer.py 80.7% <0%> (-1.76%) ⬇️
synapse/handlers/search.py 80.24% <0%> (ø) ⬆️
synapse/handlers/federation.py 61.72% <0%> (ø) ⬆️
synapse/state/v1.py 92.24% <0%> (+1.55%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update c26f49a...cfa8f6f. Read the comment docs.

@hawkowl hawkowl requested a review from a team November 27, 2018 06:15
synapse/http/client.py Outdated Show resolved Hide resolved
synapse/http/client.py Outdated Show resolved Hide resolved
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is not a given that getaddrinfo's results are cached - in general each separate call can result in a separate request to the DNS server. I'm pretty sure that with a short TTL and by rapidly switching an A record between internal and external addresses, you could get access to a blacklisted IP address in not-very-many attempts, so I'm really quite keen that we find a solution here that doesn't involve two separate DNS lookups.

synapse/rest/media/v1/preview_url_resource.py Outdated Show resolved Hide resolved
synapse/http/client.py Show resolved Hide resolved
synapse/http/client.py Outdated Show resolved Hide resolved
synapse/http/client.py Outdated Show resolved Hide resolved
synapse/http/client.py Outdated Show resolved Hide resolved
synapse/http/client.py Outdated Show resolved Hide resolved
@hawkowl
Copy link
Contributor Author

hawkowl commented Nov 30, 2018

Further looks at the IP blacklist to be done in #4242 -- reducing the scope of this PR to just the listed bug.

richvdh
richvdh previously requested changes Dec 3, 2018
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

sorry, this still feels like it makes the blacklist feature useless.

@hawkowl hawkowl requested a review from a team December 14, 2018 15:07
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Generally lgtm. A couple of points below.

@@ -151,6 +155,12 @@ def read_config(self, config):
except ImportError:
raise ConfigError(MISSING_LXML)

try:
import hyperlink
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we'll need to document this in the README (grep for lxml - there's more than one place) as well as the upgrade notes, and put something in the changelog about it, to stop people getting surprised by it suddenly not working.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This should only affect people on Twisteds older than 17.5, which hard depends on hyperlink. (I guess this affects xenial and so?) Maybe I should add it to our pip dependencies?

)
request_deferred = timeout_deferred(
request_deferred, 60, self.hs.get_reactor(),
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

all these unrelated formatting changes make this a much bigger diff, so make it much harder to review. pleeeease can we keep formatting and functional changes separate in future?

synapse/http/client.py Outdated Show resolved Hide resolved
synapse/http/client.py Outdated Show resolved Hide resolved
synapse/http/client.py Outdated Show resolved Hide resolved
synapse/http/client.py Outdated Show resolved Hide resolved
synapse/http/client.py Show resolved Hide resolved
synapse/http/client.py Outdated Show resolved Hide resolved
synapse/rest/media/v1/preview_url_resource.py Outdated Show resolved Hide resolved
@hawkowl hawkowl requested a review from a team December 20, 2018 10:01
Copy link
Member

@richvdh richvdh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

still a bit confused about BlacklistingAgentWrapper but lgtm otherwise.

def __init__(self, agent, reactor, whitelist=None, blacklist=None):
"""
An Agent wrapper which will prevent access to IP addresses being accessed
directly (without an IP address lookup).
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not really clear what you mean by "an IP address lookup" here.

Is this just an optimisation? I'm not sure the extra complexity it introduces makes it worthwhile if so.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Accessing an IP address directly means that it doesn't do a DNS lookup for said IP address.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

right, so it never hits the IPBlacklistingResolver? Is this not a problem if we get a 302 redirect to an IP address?

@hawkowl hawkowl merged commit ea6abf6 into develop Dec 21, 2018
@hawkowl hawkowl deleted the hawkowl/ip-preview branch December 21, 2018 14:56
@hawkowl
Copy link
Contributor Author

hawkowl commented Dec 21, 2018 via email

@hawkowl
Copy link
Contributor Author

hawkowl commented Dec 21, 2018 via email

@richvdh
Copy link
Member

richvdh commented Dec 21, 2018

It would make me feel better, but if you're confident it works right, I don't mind too much :)

richvdh added a commit that referenced this pull request Jan 8, 2019
Synapse 0.34.1rc1 (2019-01-08)
==============================

Features
--------

- Special-case a support user for use in verifying behaviour of a given server. The support user does not appear in user directory or monthly active user counts. ([\#4141](#4141), [\#4344](#4344))
- Support for serving .well-known files ([\#4262](#4262))
- Rework SAML2 authentication ([\#4265](#4265), [\#4267](#4267))
- SAML2 authentication: Initialise user display name from SAML2 data ([\#4272](#4272))
- Synapse can now have its conditional/extra dependencies installed by pip. This functionality can be used by using `pip install matrix-synapse[feature]`, where feature is a comma separated list with the possible values `email.enable_notifs`, `matrix-synapse-ldap3`, `postgres`, `resources.consent`, `saml2`, `url_preview`, and `test`. If you want to install all optional dependencies, you can use "all" instead. ([\#4298](#4298), [\#4325](#4325), [\#4327](#4327))
- Add routes for reading account data. ([\#4303](#4303))
- Add opt-in support for v2 rooms ([\#4307](#4307))
- Add a script to generate a clean config file ([\#4315](#4315))
- Return server data in /login response ([\#4319](#4319))

Bugfixes
--------

- Fix contains_url check to be consistent with other instances in code-base and check that value is an instance of string. ([\#3405](#3405))
- Fix CAS login when username is not valid in an MXID ([\#4264](#4264))
- Send CORS headers for /media/config ([\#4279](#4279))
- Add 'sandbox' to CSP for media reprository ([\#4284](#4284))
- Make the new landing page prettier. ([\#4294](#4294))
- Fix deleting E2E room keys when using old SQLite versions. ([\#4295](#4295))
- The metric synapse_admin_mau:current previously did not update when config.mau_stats_only was set to True ([\#4305](#4305))
- Fixed per-room account data filters ([\#4309](#4309))
- Fix indentation in default config ([\#4313](#4313))
- Fix synapse:latest docker upload ([\#4316](#4316))
- Fix test_metric.py compatibility with prometheus_client 0.5. Contributed by Maarten de Vries <[email protected]>. ([\#4317](#4317))
- Avoid packaging _trial_temp directory in -py3 debian packages ([\#4326](#4326))
- Check jinja version for consent resource ([\#4327](#4327))
- fix NPE in /messages by checking if all events were filtered out ([\#4330](#4330))
- Fix `python -m synapse.config` on Python 3. ([\#4356](#4356))

Deprecations and Removals
-------------------------

- Remove the deprecated v1/register API on Python 2. It was never ported to Python 3. ([\#4334](#4334))

Internal Changes
----------------

- Getting URL previews of IP addresses no longer fails on Python 3. ([\#4215](#4215))
- drop undocumented dependency on dateutil ([\#4266](#4266))
- Update the example systemd config to use a virtualenv ([\#4273](#4273))
- Update link to kernel DCO guide ([\#4274](#4274))
- Make isort tox check print diff when it fails ([\#4283](#4283))
- Log room_id in Unknown room errors ([\#4297](#4297))
- Documentation improvements for coturn setup. Contributed by Krithin Sitaram. ([\#4333](#4333))
- Update pull request template to use absolute links ([\#4341](#4341))
- Update README to not lie about required restart when updating TLS certificates ([\#4343](#4343))
- Update debian packaging for compatibility with transitional package ([\#4349](#4349))
- Fix command hint to generate a config file when trying to start without a config file ([\#4353](#4353))
- Add better logging for unexpected errors while sending transactions ([\#4358](#4358))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants