You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Currently, each table class inheriting from Model uses its own TableConnection instance, which has its own low-level API Connection. I have 20 table classes inheriting from Model. When using these with Lambda, I found that each table using its own Connection leads to two problems:
Initial Connection Time: Each table requires instantiating and using a Connection for the first time, which takes about 0.6 seconds. If an interface calls multiple tables for data aggregation during the first call, it results in significant waiting time.
Memory Usage: Memory usage increases with the number of table classes used. This issue memory usage growing as we add models #876 (comment) raises the same problem. I conducted a test where 20 table classes were used simultaneously, resulting in a memory usage of 216 MB. When all table classes shared a single connection, memory usage was 78 MB. This means each connection uses about 7 MB of memory. I will provide the test code below to ensure that my testing method did not lead to incorrect conclusions.
My primary execution environment is AWS Lambda, where the cost is related to both memory usage and execution time. Therefore, these two issues nearly double the execution cost of my Lambda functions.
I am considering whether all table classes can share a single low-level API Connection to address these issues.
Solution
My approach focuses on the __init__ method of TableConnection. As each table class has its own TableConnection instance, and each TableConnection has its own low-level API Connection, we can specify a shared Connection during TableConnection initialization.
The original TableConnection initialization is as follows:
Additionally, the MetaProtocol should include a connection attribute, and the Model class's _get_connection class method should pass cls.Meta.connection when instantiating TableConnection.
Problem Statement
Currently, each table class inheriting from
Model
uses its ownTableConnection
instance, which has its own low-level APIConnection
. I have 20 table classes inheriting fromModel
. When using these with Lambda, I found that each table using its ownConnection
leads to two problems:Connection
for the first time, which takes about 0.6 seconds. If an interface calls multiple tables for data aggregation during the first call, it results in significant waiting time.My primary execution environment is AWS Lambda, where the cost is related to both memory usage and execution time. Therefore, these two issues nearly double the execution cost of my Lambda functions.
I am considering whether all table classes can share a single low-level API
Connection
to address these issues.Solution
My approach focuses on the
__init__
method ofTableConnection
. As each table class has its ownTableConnection
instance, and eachTableConnection
has its own low-level APIConnection
, we can specify a sharedConnection
duringTableConnection
initialization.The original
TableConnection
initialization is as follows:I currently use a patch to solve this problem. The code is as follows:
This function needs to be called globally once.
However, I recommend directly supporting a
connection
attribute in the Meta class of pynamodb table classes. The logic is as follows:Additionally, the
MetaProtocol
should include aconnection
attribute, and theModel
class's_get_connection
class method should passcls.Meta.connection
when instantiatingTableConnection
.Testing
The test program runs on AWS Lambda.
The test code is as follows:
The following is the output when running directly:
The output after calling
_patch_pynamodb_connection
:We can see that the results are significant, with both time and memory usage being reduced.
However, I am not entirely sure if this approach might cause some logical issues, which is why I am raising this issue.
The text was updated successfully, but these errors were encountered: