Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix slave's binlog offset greater than master bug #1682

Merged
merged 10 commits into from
Jul 9, 2023

Conversation

luky116
Copy link
Collaborator

@luky116 luky116 commented Jul 4, 2023

fix #1681

@@ -138,6 +138,7 @@ func (s *Router) isOnline() bool {
func (s *Router) dispatch(r *Request) error {
hkey := getHashKey(r.Multi, r.OpStr)
var id = Hash(hkey) % uint32(models.GetMaxSlotNum())
r.HashID = id
slot := &s.slots[id]
return slot.forward(r, hkey)
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The provided code patch seems to be a small change in the dispatch method of a Router struct. The change adds a line that assigns the value of id to r.HashID. Here's a brief code review:

  1. Variable Naming: The variable id used for storing the result of Hash(hkey) % uint32(models.GetMaxSlotNum()) could benefit from a more descriptive name to improve code readability. Consider using a name that reflects the purpose or meaning of this value.

  2. Error Handling: The code snippet does not contain any error handling logic. It's important to handle any potential errors from operations like Hash() and models.GetMaxSlotNum(). Depending on your requirements, you may need appropriate error checks and error propagation mechanisms.

  3. Contextual Information: Without additional context, it's difficult to determine if this code patch is part of a larger program or system. Ensure that this change aligns with the overall design and architecture of the software.

Apart from these points, the code patch appears to be straightforward and efficient.

@@ -4,9 +4,6 @@
package proxy

import (
"sync"
"unsafe"

"pika/codis/v2/pkg/proxy/redis"
"pika/codis/v2/pkg/utils/sync2/atomic2"
)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the provided code patch, it seems that some import statements have been removed: "sync" and "unsafe". It's difficult to perform a comprehensive code review without the full context of the codebase, but here are a few general suggestions:

  1. Check if removing the "sync" and "unsafe" packages was intentional. These packages provide synchronization primitives and low-level memory access, respectively. Make sure their removal does not introduce any issues or break functionality.

  2. Review the changes made to the "import" block. Verify that the remaining imports are still necessary and relevant.

  3. Consider adding comments or documentation to explain the purpose and functionality of the package and individual functions, especially if they are non-trivial.

  4. Pay attention to error handling. Ensure that appropriate error checks and handling are implemented throughout the codebase.

  5. It's good practice to follow consistent formatting and coding conventions, such as indentation, naming conventions, and code organization, to improve code readability and maintainability.

  6. Review any logic or algorithm changes introduced in the patch. Verify that they are correct and handle all possible edge cases.

Remember, without more information about the surrounding code, it's challenging to provide specific recommendations. It's always useful to test the modified code thoroughly to identify any potential bugs or performance issues.

@OpenAtomFoundation OpenAtomFoundation deleted a comment from cr-gpt bot Jul 4, 2023
(keys[0].compare(0, KSlotKeyPrefixLen, SlotKeyPrefix) == 0 || keys[0].compare(0, KSlotKeyPrefixLen, SlotTagPrefix) == 0)) {
return Status::OK();
}

LogOffset log_offset;

stable_logger_->Logger()->Lock();
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall, the code patch you provided seems fine. Here are a few observations and suggestions:

  1. The added code segment checks if the command name is "SAdd" and if the first key in the command starts with either "SlotKeyPrefix" or "SlotTagPrefix". If these conditions are met, it returns Status::OK() without further processing. This logic assumes that kCmdNameSAdd, KSlotKeyPrefixLen, and SlotKeyPrefix/SlotTagPrefix are correctly defined and initialized elsewhere in the code.

  2. One improvement suggestion for the added code segment is to use std::string::starts_with function (introduced in C++20) instead of the compare() method. It provides a cleaner and more expressive way to check if a string starts with a specific prefix.

Example:

if (cmd_ptr->name() == kCmdNameSAdd && !keys.empty() &&
    (keys[0].starts_with(SlotKeyPrefix) || keys[0].starts_with(SlotTagPrefix))) {
  return Status::OK();
}
  1. Make sure that the LogOffset log_offset variable is being used correctly after this code block. It seems it might be used later, so double-check its usage in the surrounding code for correctness.

  2. Consider adding comments to describe the purpose and rationale of the added code segment. Comments can improve code clarity and make it easier for future maintainers to understand the code's intent.

Remember to thoroughly test the code with various scenarios to ensure it functions as intended and doesn't introduce any unintended side effects.

std::vector<std::string> keys = cmd_ptr->current_key();
// slotkey shouldn't add binlog
if (cmd_ptr->name() == kCmdNameSAdd && !keys.empty() &&
(keys[0].compare(0, KSlotKeyPrefixLen, SlotKeyPrefix) == 0 || keys[0].compare(0, KSlotKeyPrefixLen, SlotTagPrefix) == 0)) {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

key_type == "s" && ((*iter).find(SlotKeyPrefix) != std::string::npos || (*iter).find(SlotTagPrefix) != std::string::npos)
别用上面那样写

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

这个没问题,代码中大量这种写法

@@ -11,6 +11,8 @@

const std::string SlotKeyPrefix = "_internal:slotkey:4migrate:";
const std::string SlotTagPrefix = "_internal:slottag:4migrate:";
const size_t KSlotKeyPrefixLen = 27;
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

别这么写

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

金鸽的意思是,把 size_t 改为int,大家评估下

Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不是,27怎么来的?

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不是,27怎么来的?

SlotTagPrefix 字符串的长度

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

不是,27怎么来的?

优化了下,直接使用 length

@OpenAtomFoundation OpenAtomFoundation deleted a comment from cr-gpt bot Jul 4, 2023
chejinge
chejinge previously approved these changes Jul 4, 2023
@@ -4,7 +4,7 @@
package proxy

import (
"sync"
"sync"
"unsafe"

"pika/codis/v2/pkg/proxy/redis"
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Below is a code review of the provided code patch:

  • The import statement for the "sync" package has an indentation inconsistency. It should be indented with a tab or spaces to match the other imports.
  • The comment lines at the beginning of the file look incorrect. They should start with // instead of @@.
  • There is an unused import for "pika/codis/v2/pkg/proxy/redis". If it is not required, it can be safely removed.

Improvement suggestions:

  • It's important to provide more context and details about the code being reviewed in order to identify potential bug risks or improvements accurately.
  • Include the entire code snippet or function if possible, so that I can analyze it more comprehensively.

Please provide more specific information or code if you'd like a more detailed review or have any particular concerns.

@@ -32,7 +34,6 @@ std::string GetSlotKey(int slot);
std::string GetSlotsTagKey(uint32_t crc);
int GetSlotsID(const std::string &str, uint32_t *pcrc, int *phastag);
void RemSlotKeyByType(const std::string &type, const std::string &key, const std::shared_ptr<Slot>& slot);
void WriteSAddToBinlog(const std::string &key, const std::string &value, const std::shared_ptr<Slot>& slot);

class PikaMigrate {
public:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the code patch you provided, here's a brief code review:

  1. It seems like kSlotKeyPrefixLen is introduced as a constant to store the length of SlotKeyPrefix.

  2. WriteSAddToBinlog() function has been removed or commented out. Ensure that this change doesn't affect the intended functionality. Verify if this function is no longer needed or if it should be replaced with an alternative implementation.

Apart from these specific changes, without the context of the entire codebase and its purpose, it's challenging to provide further bug risks or improvement suggestions. A comprehensive code review typically involves examining the logic, error handling, performance optimizations, coding conventions, and adherence to best practices. Consider providing more details or sharing the rest of the code if you require a more thorough review.

@@ -31,7 +33,6 @@ std::string GetSlotKey(int slot);
std::string GetSlotsTagKey(uint32_t crc);
int GetSlotsID(const std::string &str, uint32_t *pcrc, int *phastag);
void RemSlotKeyByType(const std::string &type, const std::string &key, const std::shared_ptr<Slot>& slot);
void WriteSAddToBinlog(const std::string &key, const std::string &value, const std::shared_ptr<Slot>& slot);

class PikaMigrate {
public:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Here are some observations and suggestions for code review:

  1. Inconsistent formatting: The indentation of the code seems to be inconsistent. It's important to maintain a consistent coding style for better readability.

  2. Magic numbers: The value "27" appears in the code as the length of SlotKeyPrefix. Consider defining it as a constant, similar to kSlotKeyPrefixLen, to improve code clarity.

  3. Unused function: The function WriteSAddToBinlog is declared but not defined or used in the provided code patch. Make sure it is necessary and remove it if it's not needed.

  4. Missing function details: It would be helpful to see the implementation and usage of the functions GetSlotKey(), GetSlotsTagKey(), GetSlotsID(), RemSlotKeyByType(), and the class PikaMigrate to perform a more comprehensive code review.

  5. Commented out code: There doesn't seem to be any commented-out code in the provided patch, but it is good practice to remove any commented-out code from the final codebase.

Note: Without the complete code context and implementation details, it may be challenging to identify potential bugs or provide further improvement suggestions.

chejinge
chejinge previously approved these changes Jul 5, 2023
g_pika_rm->GetSyncMasterSlotByName(SlotInfo(slot->GetDBName(), slot->GetSlotID()));
Status s = sync_slot->ConsensusProposeLog(cmd_ptr);
if (!s.ok()) {
LOG(ERROR) << "write sadd key to binlog failed, key: " << key;
}
}

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the provided code patch, here are some observations and suggestions:

  1. It appears that the function WriteSAddToBinlog has been removed from the code. If this is intentional and no longer required, then the removal is fine. Otherwise, double-check if it should be kept or if there's an alternative implementation planned.

  2. There doesn't seem to be any bug risk in the modified code section since you have removed the WriteSAddToBinlog function call along with its log output. However, without the full context of the codebase, it's challenging to assess the overall impact of these changes.

  3. Consider refactoring the code to adhere to consistent coding conventions for better readability. For example, use meaningful parameter names instead of single-letter variable names like type, key, etc., to make the code more understandable.

  4. Ensure that error handling is appropriate in the calling function. Since the call to WriteSAddToBinlog has been removed, it's essential to handle errors or perform any alternative actions if necessary.

  5. Review the surrounding code and unit tests to verify that these changes do not introduce any unwanted side effects or break existing functionality.

  6. It's generally a good practice to include comments explaining the purpose and intention behind each code change to provide clear documentation for future developers.

Remember to thoroughly test the modified code to ensure its correctness and robustness after making any changes.

chejinge
chejinge previously approved these changes Jul 5, 2023
wangshao1
wangshao1 previously approved these changes Jul 5, 2023
@luky116 luky116 dismissed stale reviews from wangshao1 and chejinge via 56a1e18 July 6, 2023 05:42
@@ -31,7 +32,6 @@ std::string GetSlotKey(int slot);
std::string GetSlotsTagKey(uint32_t crc);
int GetSlotsID(const std::string &str, uint32_t *pcrc, int *phastag);
void RemSlotKeyByType(const std::string &type, const std::string &key, const std::shared_ptr<Slot>& slot);
void WriteSAddToBinlog(const std::string &key, const std::string &value, const std::shared_ptr<Slot>& slot);

class PikaMigrate {
public:
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Without the complete context and understanding of the codebase, it is challenging to provide a thorough code review. However, based on the provided code patch, here are some observations:

  1. Unused Function: The function WriteSAddToBinlog seems to be declared but not used in the code patch. If it is not necessary, consider removing it or verifying if it serves any purpose.

  2. Magic Numbers: The value 10 * 1024 appears as MaxKeySendSize. Consider using named constants or variables with descriptive names instead of "magic numbers" to improve code readability and maintainability.

  3. Style Consistency: Ensure consistent code formatting and indentation throughout the codebase for better readability and maintainability.

Apart from these specific observations, perform a thorough testing of the modified code to check for bug risks or potential improvements. Additionally, consider including comments that explain the purpose and functionality of key functions or blocks of code to aid future maintenance.

@wanghenshui wanghenshui self-requested a review July 7, 2023 03:09
wanghenshui
wanghenshui previously approved these changes Jul 7, 2023
Copy link
Collaborator

@wanghenshui wanghenshui left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

没意见。文档说明一下特殊key的处理。能测的话加一下测试用例

cp ../conf/pika.conf ./pika_master.conf
cp ../conf/pika.conf ./pika_slave.conf
cp ../tests/conf/pika.conf ./pika_master.conf
cp ../tests/conf/pika.conf ./pika_slave.conf
mkdir slave_data
sed -i -e 's|databases : 1|databases : 2|' -e 's|#daemonize : yes|daemonize : yes|' ./pika_master.conf
sed -i -e 's|databases : 1|databases : 2|' -e 's|port : 9221|port : 9231|' -e 's|log-path : ./log/|log-path : ./slave_data/log/|' -e 's|db-path : ./db/|db-path : ./slave_data/db/|' -e 's|dump-path : ./dump/|dump-path : ./slave_data/dump/|' -e 's|pidfile : ./pika.pid|pidfile : ./slave_data/pika.pid|' -e 's|db-sync-path : ./dbsync/|db-sync-path : ./slave_data/dbsync/|' -e 's|#daemonize : yes|daemonize : yes|' ./pika_slave.conf
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the code patch you provided, here are some observations and recommendations for improvement:

  1. File Paths:

    • The script is copying the pika.conf file from either ../conf/ or ../tests/conf/ to the current directory. Make sure these paths exist and point to the correct location.
    • It's generally a good practice to use absolute paths when dealing with file operations to avoid any ambiguity.
  2. Script Purpose:

    • It would be helpful to add comments or documentation explaining the purpose of the script, what it does, and how it should be used. This can make it easier for others (and yourself) to understand and modify the script in the future.
  3. Code Duplication:

    • The lines cp ../tests/conf/pika.conf ./pika_master.conf and cp ../tests/conf/pika.conf ./pika_slave.conf are duplicated. Consider assigning the path to a variable once and then using that variable for both copy operations.
  4. Error Handling:

    • Currently, there is no error handling in the script. It might be beneficial to check if the copy operations (cp) are successful before proceeding further. You can consider adding conditional checks or error handling mechanisms to handle such scenarios gracefully.
  5. Consistency:

    • Review the changes made by the sed commands to ensure they are aligned with your requirements and configuration.
    • For consistency, consider using the same quoting style consistently throughout the script (either single quotes or double quotes).

Remember that a thorough code review requires a deeper understanding of the project and its specific requirements. These suggestions are based solely on the provided patch.

@luky116 luky116 merged commit e306091 into OpenAtomFoundation:unstable Jul 9, 2023
8 checks passed
bigdaronlee163 pushed a commit to bigdaronlee163/pika that referenced this pull request Jun 8, 2024
…#1682)

* fix slave's binlog offset greater than master bug
cheniujh pushed a commit to cheniujh/pika that referenced this pull request Sep 24, 2024
…#1682)

* fix slave's binlog offset greater than master bug
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

slave 节点的 binlog offset 比 master 节点大,导致 trySync 失败
5 participants