Cisco Switching/Routing :: Output Queue Drops On 3750 Core Stack And 3560 Edge Switches
Jan 10, 2013
I've been fighting what seems to be an increased number of outqueue drops on our core stack and edge switches for the last 3 or 4 weeks.(The core consists of a stack of 5 3750s in 32-gig stack mode. The wkgrp switches are 3560s. all are at 12.2.52) The wkgrp switches are directly connected to users. We use Nortel IP phones with the phone inline with the user PC. auto-neg to 100/full. [code] However I have tried turning off QOS on a couple of workgroup switches (no mls qos, but left individual port configurations the same) but am still seeing drops.Since I have disabled qos on the switches in question (no mls qos) (not the core tho) I am presuming these commands have no affect on the switch operation and therefore cannot be related to the problem. With QOS turned off one would presume that it is general congestion - especially at the user edge where busy PC issues might contribute. So I wanted to see if I could see any instances of packets in the output queues building up.
I wrote some scripts and macros that essentially did a snapshot of 'show int' every 20 seconds or so, and looked for instances of 'Queue: x/' where x was greater than zero.What I found after several days of watching the core stack, and a few of the workgroup switches that are most often displaying the behavior, was that I NEVER saw ANY packets in output queues. I often saw packets in Input queues for VLAN1, once in a great while I would see packets on input queues for fa or Gi interfaces, but NEVER on output queues. [ code] Additionally, when I look (via snmp) at interface utilization on interfaces showing queue drops (both core and wkgroup), they are occurring at ridiculously low utilization levels (as low as 4 to 8%). I've tried to look for microbursts between the core and a wkgroup switch where the core interface was experiencing drops, but haven't seen any (using observer suite). [code] While the queue-drop counts aren't critically high at this point, they are happening more frequently than in the past and I would like to understand what is going on... In most cases, no error counters are incrementing for these interfaces. Is there some mechanism besides congestion that could cause output queue drops?
I feel that 3560 and 3750 perform differently with the following two commands:
srr-queue bandwidth shape 5 0 0 0 srr-queue bandwidth limit 50 On 3750, the bandwidth for queue 1 is limited to 100mbps x 50% / 5 = 10mbps On 3560, the bandwidth for queue 1 is limited to the smaller value of BW / shape weight and BW x limit%.
Does it sound about right? is there a way to check for mls qos input queue drops? The show mls qos interface xxx stat only shows the output queue drops. Maybe for some reason the input queue never drops?
We currently have a site with a very simple topology that uses a 3750X switch stack for a collapsed core. Everyday, the users have a conference call and experience poor voice quality.Its not bad when users call from several conference phones, but when everyone calls in on individual phones, there is choppy and almost inaudible voice quality experienced. The voice traffic flow would be as follows: Phone <-> 3750 switch <-> Voice GW We have packet captures showing that RTP packet loss is occuring from the phone to the voice gateway, but none from the voice gateway to the phones. We also have drops in the output queues that match drops on the asics. I can reset the counters and they will be clear until the call, and then they increment significantly during the call. The voice gateway and phones are non-Cisco. The switch stack has 6 switches. We are trusting the DSCP settings on the phones. All the queue drops from the phones are usually in queues 0-3, but all drops on the voice gateway is in queue 0. Below are the QoS settings; they are mostly default and we have not changed any queuing, thresholds, or buffers. Should we specify larger buffers and threshold for a designated queue and send EF traffic to that queue?
MySwitch#sh mls qos QoS is enabled QoS ip packet dscp rewrite is disabled Typical Port GigabitEthernet1/0/4 trust state: trust dscp
I configure 3750 stack switch as core and 2960 stack switches as access layer switches.I connected my laptop to one of my core stack in VLAN 10 and I am pinging to one of my server in VLAN 1. What will be the minimum latency at the time of inter VALN routing
we have two 6509E, as our core switches. Recently I noticed that on some connections I have a high output queue drop rate.
These 4 x 2 interfaces (gigabit) are connected to our blade encolure, consisting of 4 x WS-CBS3120X-S. The utilization of the links is really quite low, when I see the increase of the drops. (~=60Mbps). All the links are fiber (SFP) and the distance between the core switches and the enclosure is about 15-20m.
I am not aware of any service degradation on the part of the servers. No CRCs, collisions etc, on the interfaces, apart from the drops.
The line card is a WS-X6748-SFP, but other interfaces don't seem to be experiencing any problems.
We're having kind of a problem with our Catalyst 4507r switches. If we do a "show interface" command we're getting a lot of "Total output drops" on some of our interfaces. It seems to be most of the time on the same vlan.I was wondering if it has got something to do with QOS or queue selection As we don't have any QOS markings configured, is it possible that all traffic is using only one of the four tx queue's?
I've been working on breaking down and understanding the default auto qos configuration on a Cisco 3750 in the hopes of putting together a QoS strategy that will fit our environment. I'm having some difficulty understanding how the "mls qos queue-set output" syntax works.
From another post, at [URL], the author offers the following example and explanation;
How come there is syntax stating "threshold 2" when in the succeeding part the 400 refers to thresshold 1 and threshold 2 again? The syntax 400 400 is, apparently, already referring to thresshold 1 and 2, no?
When configuring QoS on 3750s/3560s, we're mapping packets to particular interface output queues with commands such as: [code] The command to see what's actually being enqueued, dropped, etc. is: [code]
Note that these queues are numbered 0 - 3, and not 1 - 4. We've been assuming that the first queue number in the "mls qos" (i.e., 1) command maps to the first queue (i.e., 0) in the "show mls qos" command.
I have a very basic networking question If I have, say, 3750's (or any L3 switch, capable of routing) at the edge and a 4500 at the core, where should I route? At the edge? At the Core? Both?
I am seeing Interface output drops that appera to be incorrect. When I do "Show Interface gi1/0/20", I will get interface output drops of "4294961382". But, when I do the same command again it shows "0" drops. Is this a reporting error? I am ruinning c3750-ipservicesk9-mz.122-58.SE.bin on a 3750 stack with 2 switches in the stack. [code]
One of my Catalyst 3750 switch have many out drops, I execute "sh mls qos int g2/0/3 statist" command, there are many output drops in queue3 threshold3. [code]
I have a 3750X-24T in our production environment that is showing very high number of OQD's in the 'show int sum' output for 4 of the Gigabit interfaces; the interfaces are each in a seperate port channel and there are no OQD's for the relevant port channel and there are no output drops showing in the output for the 'sh int' command for each interface.
The following are the OQD's for the relenvant interfaces Gi1/1/1: 0 Gi1/1/2: 0 Gi1/1/3: 0 Gi1/1/4: 0 Gi2/1/1: 4252879251 Gi2/1/2: 4251090833 Gi2/1/3: 4251754140 Gi2/1/4: 4294942102 Po1: 0 Po2: 0 Po3: 0 P04: 0
Gi1/1/1 and Gi2/1/2 assigned to Po1, and so on. IOS version: C3750E-IPBASEK9-M 12.2(58)SE2
If I have two stackable switches one switch uplinks to one 6509 core switch and the other connection uplinks to another 6509 core switch, and also the other stackable switch does not connect to the core switches. Because I am using hsrp and also we are not using vss then one uplink to the core is not in used only ones is so then how does creating an etherchannel between does two uplinks to both core switches benefit me in anyway such as more bandwith and using both uplinks at the same time or I'm I wrong?
If I have two stackable switches were only one stackable switch has two uplinks one uplink goes to one core 6509 switch and the other uplink goes to the other 6509 core switch can a Layer 3 etherchannel be used if each uplink go to a different core switch, by the way hsrp is running between both switches and also can you give an example how data will be routed from the stackable switch through the ethernetchannel to one of the core switch accross the WAN to another core switch?
The company I work have finally decided to enter the 21st century and invest in a new telephone system (Interactive Intelligence) to replace the legacy system which has served us well for the past 10 years. The project has only just started and involves upgrading sections of CAT3 cabling to CAT6, replacing Cisco 3550 switches in one area of the building with Cisco 4507 switches and upgrading our Core switches with Cisco Nexus 7010's. The area that concerns me most is enabling the network for qos as I have very little experience with it. At the moment Im trying to read as much documentation as I can on QOS to bring myself up to speed.
The access layer switches will consist of a mixture of Cisco 3750 & 4507 switches connected to Cisco Nexus 7010 switches which will form a collapsed aggregation & core layer.
Basically, how I should approach this daunting task of making sure the network will support VOIP.
Looking for feedback from other organizations that have large 3750 stacks. I've got one stack of (8) 3750's composed of (6) 3750G's and (2) plain 3750's. This particular stack is usually unresponsive to SNMP queries and often fails to write config when we make changes. After a couple tries it will finally go. Part of my probably here is likely the plain 3750's that always boot faster and come up as the master. I should manually set the master to one of the G's. What I'm wondering is who else has 7-9 3750's stacked and are they performing well for SNMP, telnet, etc? I've got another newer stack of 7 3750E's that I need to add one more switch to. Need to decide if I want extended downtime to break the stack up or just add the 3750X to make member 8 and hope it performs well. I have 50+ 3750 stacks working great on our campus.
I am trying to do ios upgrade on 5 stacked 3750 switches. All the switches have different model number, so i am wondering which image file i should download. As far as i understand all the stacked switches should have the same IOS, i may be wrong. The switches have the following model numbers and SW images;
We have several stacks of 3750 switches where this problem is occurring. All of the switches are running IOS version 12.2(50)SE3. (Yes, I know it's old.)
We're seeing inconsistency in how the stack reports members when issuing the following commands:
sow switch show version show inventory
Not all members are showing up in the output of the "show version" and "show inventory" commands. For example:
HUNTI-WV-WDAC03#sh switch H/W Current Switch# Role Mac Address Priority Version State ---------------------------------------------------------- 1 Member 108c.cff2.ea00 15 0 Ready 2 Member 8cb6.4f0a.a600 1 0 Ready *3 Master 8cb6.4f0a.9c00 1 0 Ready (all three switches in stack are displayed)(code)
I am wanting to etherchannel from a 3750 stack to core Layer 3 switches (also 3750) with a cable going to each core switch, I have put both core switches and the stack under a 28bit subnet mask, but I dont seem to be load balancing across both links.
I have 3 3750 staking switch with the following configuration:
H/W Current Switch# Role Mac Address Priority Version State ---------------------------------------------------------- 1 Member 442b.0350.9400 1 1 Ready 2 Member 442b.0357.8780 2 1 Ready 3 Master 442b.036f.a800 3 1 Ready
The new switch is installed to be wired to the master switch. the swtich 3 (* 3 Master Ready 442b.036f.a800)my question is: if you disconnect the wiring from the master stack to connect to the new switch, it is possible that the new switch change the current configuration of stacking.
-Can I just add the new switch with clean configuration without changing the current configuration ? -How is assigned the stack number? i need the new switch be with the GI4 / x?. I can I manually by assigning ports? -If I want to the new 3750 member witch lower prioridad, can I change priority before adding to stack?
I have been given 2 x 3750 switches that were a stack, I need to keep the configs on both but how do I make them individual switches again and remove the stack info?
Lets say i have 2 3750 switches stacked via backend stack cables. Now if a packet needs to go from 1 switch in the stack to second switch in the stack, will it travel via stack cable or do we need to connect both switches via uplink ports (ethernet or sfp). I tried reading datasheet but it no wheres mention the actual frame path between switches in stack.
I'm thinking about using a 3750 stacking around 4 or 5 switches intead of using one 4500 series switch.
1-My first argument was concerning the budget (one 4500 instead of four or five 3750 in stack mode) ---- Here i think 4 or 5 stacked 3750 is cheap than one 4500 2- My second argument was about harware redundancing that i don't have with one 4500 and that i have with stack mode ----Here i have more hardware redundancing with 3750 3- The bug that we notice today in 2960 and 3750 and not in 4500 4- Simple to manage a 4500 compare to 3750 stack design 5- I was looking also for the one with best backplane ----Here i didn't have any infos concerning the best backplane between 4 or 5 stack 3750 and a 4500. 6- I'm also looking for power redundancing.
I have an issue with a Cisco 3750 switch stack which is connected to two seperate upstream Cisco 3750's which are administered by an ISP. The ISP is experiencing MAC address flapping from one of my VLAN SVI's i am using to route traffic upstream
[code]...
As you can see I utilise a VLAN SVI to route traffic to the upstream 1.1.1.2 (illustration only) IP. As per the diagram of the topology attached, the ISP is receiving a MAC address flapping error confirming the SVI MAC address from my switch stack is being learnt on the trunk port connecting switch 1 and switch 2, and also the port (Gi1/0/48) directly connected to my switch stack. As these are all Layer 2 links essentially being passed upstream and then connected between the two ISP switches, we have a 3 way triangular loop formed.
If I was to remove the port channel configuration from the two ports associated with the VLAN SVI, am i right in suggesting this would still form a layer 2 loop? The two ports would still be a member of the SVI VLAN, and it is the VLAN MAC address which is being learnt by the two ISP switches on different interfaces.
We have a couple of Cisco switches and connected a (Windows 7) laptop to one of them and it gets its IP address from a DHCP server.I can now ping the IP from all of the switches, no problem, also not when I log on to the core switch in the same VLAN as both notebooks. But from my (Windows 7) laptop, which is in the same VLAN as the target laptop, I cannot ping it.
I checked, default gateway is good on both sides, as are DNS servers.
I have a 3750 as a core and have a series of HP Procurve switches that are daisy chained using one port. I have two vlans on the port now (6 &9) and everything works fine, all switches communicate and end devices on the switches are also talking. There is a requirement to add a device towards the end of the chain which requires it to connect using Vlan1. Once I add Vlan1 to the port onthe 3750 I lose connectivity to all the HP switches.
We're having some issues with a 3560 CPE. It's uplink is a GE fiber link, customer port is FE RJ45. We see a lot of TX-frames being dropped at the FE port,but none at the GE port. Even when the customer is only at ~50Mbit/s downstream traffic. When customer sending ~50Mbit/s upstream, there are no TX- drops at the GE link. Is this a normal behaviour? From what I know the physical medium shouldn't have any inpact on this since drops occurs in the port- asic, and not in physical transmission.Do the buffer-sizes between GE and FE differ? What could we do to optimize the flow and reduce drops? QoS is set to off and no modifications on the queues have been made on the interfaces.
I can see drops on the 6509 Queue for interface gi1/6 , qos is disabled globaly with qos disabled all packets are in one Queue using best effort my question is if I can see drops using the sh queueing int Gi1/6 command why I am not seeing any drops when I run the Sh int (interface number ) command. [code]
I can see drops on one of our busy L3 vlan in the input queue and are going up very frequently.System image file is "sup-bootflash:s72033-psv-mz.122-18.SXD3.binHardware = 6509
i have a 3560 connecting to a sp with limited bandwidth. i have one interface on the switch whose traffic i do not want to drop. i want this traffic to go into the high priority queue. i am not sure how this should be configured, but here is my best guess and my current qos configuration on the switch:
I am working on a QoS design which I hope to test at some point, but at this stage its from the books.My question is how to decide which queue and threshold to use for video traffic, then lower priority traffic.I understand the shaping and sharing commands, its the queuing and threshold bit I'm not clear on.The plan is to use the priority-queue for EF marked voice, this will be policed on ingress to provide an upper limit to EF traffic levels, then my second priority traffic will be video. Which queue will get serviced first once the priority queue is empty, and how do I decide which threshold to allocate my video traffic to? The document ion is not at all clear, I want to prioritse my traffic in the following order:
1 voice, use the priority queue 2 video, this to get serviced ahead of data, after voice. 3 interactive data 4 Bulk data 5 Best effort
So Q1 settings are ignored due to priority queue. Q2 gets 70%, Q3 25% etc.Is it as simple as putting video into Q2 T1, then interactive data into Q2 T2, will Q2T1 get a higher priority over Q2 T2 once the PQ is serviced?
After opening up Solarwinds NPM, I noticed that a few of my interfaces had lots of discards (who knows how long it's been sets the counters were reset)
interface GigabitEthernet1/0/25description Etherchannel to MamaCassswitchport trunk encapsulation dot1qswitchport mode trunkswitchport nonegotiatepriority-queue outchannel-group 4 mode on
interface GigabitEthernet2/0/25description Etherchannel to MamaCassswitchport trunk encapsulation dot1qswitchport mode trunkswitchport nonegotiatepriority-queue outchannel-group 4 mode on
interface Port-channel4switchport trunk encapsulation dot1qswitchport mode trunkswitchport nonegotiate,It looks as if priority-queue was configured outbound on these interfaces, could this be the cause of the transmit discards which are now up to 79,835, I just reset the counters on the interfaces a little while ago.
I'm not the best in the world when it comes to QoS, we do have some VoIP phones, but they are only a specific network, and do not travel outside, since there are used mainly for VoIP training. I do know both interfaces are running the default of FIFO.
