[snmp] test_snmp_default_route failure

[qiluo-msft]

Description

Steps to reproduce the issue:

Describe the results you received:

Error Message
KeyError: 'snmp_cidr_route'
Stacktrace
duthosts = <tests.common.devices.duthosts.DutHosts object at 0x7f8838de7a10>
enum_rand_one_per_hwsku_frontend_hostname = 'str2-7050cx3-acs-01'
localhost = <tests.common.devices.local.Localhost object at 0x7f8836fbb950>
creds_all_duts = {<MultiAsicSonicHost> str2-7050cx3-acs-01: {'ASIC0': {'configuration': {'ASIC4': {'bgp': {'asn': 65100, 'peers': {...}...dut_asn': 65100}}, 'topology': {'NEIGH_ASIC': {'ASIC4': {'asic_intfs': [...]}, 'ASIC5': {'asic_intfs': [...]}}}}, ...}}

    @pytest.mark.bsl
    def test_snmp_default_route(duthosts, enum_rand_one_per_hwsku_frontend_hostname, localhost, creds_all_duts):
        """compare the snmp facts between observed states and target state"""
    
        duthost = duthosts[enum_rand_one_per_hwsku_frontend_hostname]
        hostip = duthost.host.options['inventory_manager'].get_host(duthost.hostname).vars['ansible_host']
        snmp_facts = localhost.snmp_facts(host=hostip, version="v2c", community=creds_all_duts[duthost]["snmp_rocommunity"])['ansible_facts']
        dut_result = duthost.shell('show ip route 0.0.0.0/0 | grep "\*"')
    
        dut_result_nexthops = []
        # ipCidrRouteEntry MIB for default route will have entries
        # where next hop are not eth0 interface.
        for line in dut_result['stdout_lines']:
            if 'via' in line:
                ip, interface = line.split('via')
                ip = ip.strip("*, ")
                interface = interface.strip("*, ")
                if interface != "eth0":
                    dut_result_nexthops.append(ip)
    
        # If show ip route 0.0.0.0/0 has route only via eth0,
        # or has no route snmp_facts for ip_cidr_route
        # will be empty.
        if len(dut_result_nexthops) == 0:
            assert 'snmp_cidr_route' not in snmp_facts, 'snmp_cidr_route should not be present in snmp_facts'
    
        if len(dut_result_nexthops) != 0:
            # Test to ensure show ip route 0.0.0.0/0 result matches with SNMP result
            for ip in dut_result_nexthops:
>               assert ip in snmp_facts['snmp_cidr_route'], "{} ip not found in snmp_facts".format(ip)
E               KeyError: 'snmp_cidr_route'

creds_all_duts = {<MultiAsicSonicHost> str2-7050cx3-acs-01: {'ASIC0': {'configuration': {'ASIC4': {'bgp': {'asn': 65100, 'peers': {...}...dut_asn': 65100}}, 'topology': {'NEIGH_ASIC': {'ASIC4': {'asic_intfs': [...]}, 'ASIC5': {'asic_intfs': [...]}}}}, ...}}
dut_result = {'stderr_lines': [], u'cmd': u'show ip route 0.0.0.0/0 | grep "\\*"', u'end': ..., weight 1', u'  * 10.0.0.63, via PortChannel0004, weight 1'], 'failed': False}
dut_result_nexthops = ['10.0.0.57', '10.0.0.59', '10.0.0.61', '10.0.0.63']
duthost    = <MultiAsicSonicHost> str2-7050cx3-acs-01
duthosts   = <tests.common.devices.duthosts.DutHosts object at 0x7f8838de7a10>
enum_rand_one_per_hwsku_frontend_hostname = 'str2-7050cx3-acs-01'
hostip     = u'10.3.146.85'
interface  = 'PortChannel0004, weight 1'
ip         = '10.0.0.57'
line       = u'  * 10.0.0.63, via PortChannel0004, weight 1'
localhost  = <tests.common.devices.local.Localhost object at 0x7f8836fbb950>
snmp_facts = {'ansible_all_ipv4_addresses': [], 'ansible_sysCachedMemory': 1132356, 'ansible_sysTotalBuffMemory': 176472, 'ansible_sysTotalFreeMemery': 12955092, ...}

snmp/test_snmp_default_route.py:38: KeyError

Describe the results you expected:

Output of show version:

(paste your output here)

Output of show techsupport:

(paste your output here or download and attach the file here )

Additional information you deem important (e.g. issue happens only occasionally):

[qiluo-msft]

Found error message in syslog

Jul 11 12:31:15.165485 str2-7050cx3-acs-01 ERR snmp#snmp-subagent [ax_interface] ERROR: MIBUpdater.start() caught an unexpected exception during update_data()#012Traceback (most recent call last):#012  File "/usr/local/lib/python3.7/dist-packages/ax_interface/mib.py", line 43, in start#012    self.update_data()#012  File "/usr/local/lib/python3.7/dist-packages/sonic_ax_impl/mibs/ietf/rfc4292.py", line 70, in update_data#012    nexthops = ent["nexthop"]#012KeyError: 'nexthop'