From birgit at westhawk.co.uk Tue Jun 3 11:43:31 2008 From: birgit at westhawk.co.uk (Birgit Arkesteijn) Date: Tue Jun 3 10:43:39 2008 Subject: [snmp] SNMP Set Response is lost / packet dropped by Westhawk API In-Reply-To: <004d01c8c299$431b5a30$e5fc5980@7C911> References: <004d01c8c299$431b5a30$e5fc5980@7C911> Message-ID: <48451243.2040800@westhawk.co.uk> Hi, This is not good news, it shouldn't happen. I currently don't have much time to investigate, sorry, and problems like this are not easy to reproduce. Which version of the stack are you using and which SNMP version are you using? Some background: As soon as a SnmpContext sends a request, it's run() method will keep listening for incoming responses until destroy() is called. Dealing with the response is done on the receive thread. It might be possible that the context misses an incoming request, however, the context retransmits the request a couple of times. There could be a race condition. Kevin, The AsnObject class has a static method called setDebug. If you call AsnObject.setDebug(11); early on when using the stack, it will print out (amongst many other things) every message the SnmpContext receives. It won't fix your problem, but it will tell you if the stack receives the response at the lower level, but somehow not deals with it higher up. Could you look into it and let us know your findings? Thanks, Birgit On 30/05/08 22:07, Josh Bers wrote: > I too have noticed problems with set requests on the stack when attempting > to change snmpv3 passwords using the USM mib, RFC 3414... > > -----Original Message----- > From: snmp-bounces@snmp.westhawk.co.uk > [mailto:snmp-bounces@snmp.westhawk.co.uk] On Behalf Of Kevin Bond > Sent: Friday, May 30, 2008 5:37 AM > To: snmp@snmp.westhawk.co.uk > Subject: [snmp] SNMP Set Response is lost / packet dropped by Westhawk API > > > Hi, > > I have been using WestHawk SNMP API for few years now. Currently I > am facing a peculiar packet loss issue. > > I have a program that sends a series of set requests to an SNMP > device. All the set-requests work perfectly in standalone. When executed one > after another, they execute successfully 5 out of six times in the other > instance I get a Timeout exception in a Random place. > > When I attempted to trace the packets with Wireshark (assuming that it is a > device problem) I noticed that the response for the set request was properly > received by the machine immediately, but It was not captured by the > particular Westhawk session. > > I am not sure how to fine tune the WestHawk API to prevent it from dropping > any packets. Thanks, Kevin > -- -- Birgit Arkesteijn, birgit@westhawk.co.uk, -- Westhawk Ltd, Albion Wharf, 19 Albion Street, Manchester M1 5LN, UK -- Company no: 1769350 -- Registered Office: -- 15 London Road, Stockton Heath, Warrington WA4 6SJ. UK. -- tel.: +44 (0)161 237 0660 -- From jbers at bbn.com Wed Jun 4 12:35:32 2008 From: jbers at bbn.com (Josh Bers) Date: Wed Jun 4 19:36:24 2008 Subject: [snmp] SNMP Set Response is lost / packet dropped by Westhawk API In-Reply-To: <48451243.2040800@westhawk.co.uk> Message-ID: <002301c8c671$d7bc8700$12fd5980@7C911> Birgit, In trying to understand why the stack might drop some responses, there are a few questions: 1. Are set requests retried the same way as get requests? Namely if a response is not received they are resent? (this might be risky for setting things that should only be set once...) 2. If an observer for receiving PDU responses does a lot of processing, what happens to the receiver thread that is listening on the inbound socket? If answer to above is that the receiver thread and the thread that calls update on the observer are the same then there may be a problem of dropping UDP packets... Since there are no guarantees on UDP and there is a finite queue that the OS has for UDP sockets (I think 5 deep is the default), then the TCP/IP in the OS may well just drop response PDU's when there are a lot of them and the Observers are causing calls to receivePacket to be spaced out. 3. If the answer to the above is that there is a separate thread to listen for response on the socket and a separate one that dispatches to observers, then there may be a queuing problem. How does the stack queue up these received packets? Josh -----Original Message----- From: snmp-bounces@snmp.westhawk.co.uk [mailto:snmp-bounces@snmp.westhawk.co.uk] On Behalf Of Birgit Arkesteijn Sent: Tuesday, June 03, 2008 2:44 AM To: List for discussion of the Westhawk SNMP stack Subject: Re: [snmp] SNMP Set Response is lost / packet dropped by Westhawk API Hi, This is not good news, it shouldn't happen. I currently don't have much time to investigate, sorry, and problems like this are not easy to reproduce. Which version of the stack are you using and which SNMP version are you using? Some background: As soon as a SnmpContext sends a request, it's run() method will keep listening for incoming responses until destroy() is called. Dealing with the response is done on the receive thread. It might be possible that the context misses an incoming request, however, the context retransmits the request a couple of times. There could be a race condition. Kevin, The AsnObject class has a static method called setDebug. If you call AsnObject.setDebug(11); early on when using the stack, it will print out (amongst many other things) every message the SnmpContext receives. It won't fix your problem, but it will tell you if the stack receives the response at the lower level, but somehow not deals with it higher up. Could you look into it and let us know your findings? Thanks, Birgit On 30/05/08 22:07, Josh Bers wrote: > I too have noticed problems with set requests on the stack when attempting > to change snmpv3 passwords using the USM mib, RFC 3414... > > -----Original Message----- > From: snmp-bounces@snmp.westhawk.co.uk > [mailto:snmp-bounces@snmp.westhawk.co.uk] On Behalf Of Kevin Bond > Sent: Friday, May 30, 2008 5:37 AM > To: snmp@snmp.westhawk.co.uk > Subject: [snmp] SNMP Set Response is lost / packet dropped by Westhawk API > > > Hi, > > I have been using WestHawk SNMP API for few years now. Currently I > am facing a peculiar packet loss issue. > > I have a program that sends a series of set requests to an SNMP > device. All the set-requests work perfectly in standalone. When executed one > after another, they execute successfully 5 out of six times in the other > instance I get a Timeout exception in a Random place. > > When I attempted to trace the packets with Wireshark (assuming that it is a > device problem) I noticed that the response for the set request was properly > received by the machine immediately, but It was not captured by the > particular Westhawk session. > > I am not sure how to fine tune the WestHawk API to prevent it from dropping > any packets. Thanks, Kevin > -- -- Birgit Arkesteijn, birgit@westhawk.co.uk, -- Westhawk Ltd, Albion Wharf, 19 Albion Street, Manchester M1 5LN, UK -- Company no: 1769350 -- Registered Office: -- 15 London Road, Stockton Heath, Warrington WA4 6SJ. UK. -- tel.: +44 (0)161 237 0660 -- _______________________________________________ snmp mailing list snmp@snmp.westhawk.co.uk http://snmp.westhawk.co.uk/mailman/listinfo/snmp From hibond000 at gmail.com Thu Jun 5 01:39:25 2008 From: hibond000 at gmail.com (Kevin Bond) Date: Wed Jun 4 20:16:51 2008 Subject: [snmp] SNMP Set Response is lost / packet dropped by Westhawk API In-Reply-To: <002301c8c671$d7bc8700$12fd5980@7C911> References: <48451243.2040800@westhawk.co.uk> <002301c8c671$d7bc8700$12fd5980@7C911> Message-ID: <81367b5d0806041209u34ec5533uff199c9d976c5880@mail.gmail.com> Hi Birgit, Thanks for your quick response and I apologize for my delayed reply. I resolved the issue after making some changes to the frequency of SNMP V2C Context creation. I have given root cause observed to help you get a better handle on the issue. In the project, I had an SNMP Adapter layer where I used to create a Context to execute the command and destroy the context immediately after the execution. This layer had no visibility on the frequency of execution. In the particular scenario, the layer using the SNMP Adapter sent a lot of set requests in sequence (one by one). In this scenario WestHawk stack had the mentioned problem in receiving the response from the device. I changed the context handling to use the same SNMP V2c Context for subsequent requests to avoid the problem. I am using the latest 6.0 stack and only V2C Contexts. Hope this helps, Kevin On Thu, Jun 5, 2008 at 12:05 AM, Josh Bers wrote: > Birgit, > > In trying to understand why the stack might drop some responses, there are > a > few questions: > > 1. Are set requests retried the same way as get requests? Namely if a > response is not received they are resent? (this might be risky for setting > things that should only be set once...) > > 2. If an observer for receiving PDU responses does a lot of processing, > what > happens to the receiver thread that is listening on the inbound socket? > > If answer to above is that the receiver thread and the thread that calls > update on the observer are the same then there may be a problem of dropping > UDP packets... > Since there are no guarantees on UDP and there is a finite queue that the > OS > has for UDP sockets (I think 5 deep is the default), then the TCP/IP in the > OS may well just drop response PDU's when there are a lot of them and the > Observers are causing calls to receivePacket to be spaced out. > > 3. If the answer to the above is that there is a separate thread to listen > for response on the socket and a separate one that dispatches to observers, > then there may be a queuing problem. How does the stack queue up these > received packets? > > Josh > > -----Original Message----- > From: snmp-bounces@snmp.westhawk.co.uk > [mailto:snmp-bounces@snmp.westhawk.co.uk] On Behalf Of Birgit Arkesteijn > Sent: Tuesday, June 03, 2008 2:44 AM > To: List for discussion of the Westhawk SNMP stack > Subject: Re: [snmp] SNMP Set Response is lost / packet dropped by Westhawk > API > > Hi, > > This is not good news, it shouldn't happen. > I currently don't have much time to investigate, sorry, and problems > like this are not easy to reproduce. > > Which version of the stack are you using > and which SNMP version are you using? > > Some background: > As soon as a SnmpContext sends a request, it's run() method will keep > listening for incoming responses until destroy() is called. Dealing with > the response is done on the receive thread. > > It might be possible that the context misses an incoming request, > however, the context retransmits the request a couple of times. > There could be a race condition. > > > Kevin, > The AsnObject class has a static method called setDebug. > If you call > AsnObject.setDebug(11); > early on when using the stack, it will print out (amongst many other > things) every message the SnmpContext receives. > > It won't fix your problem, but it will tell you if the stack receives > the response at the lower level, but somehow not deals with it higher up. > > Could you look into it and let us know your findings? > > Thanks, Birgit > > > > On 30/05/08 22:07, Josh Bers wrote: > > I too have noticed problems with set requests on the stack when > attempting > > to change snmpv3 passwords using the USM mib, RFC 3414... > > > > -----Original Message----- > > From: snmp-bounces@snmp.westhawk.co.uk > > [mailto:snmp-bounces@snmp.westhawk.co.uk] On Behalf Of Kevin Bond > > Sent: Friday, May 30, 2008 5:37 AM > > To: snmp@snmp.westhawk.co.uk > > Subject: [snmp] SNMP Set Response is lost / packet dropped by Westhawk > API > > > > > > Hi, > > > > I have been using WestHawk SNMP API for few years now. Currently > I > > am facing a peculiar packet loss issue. > > > > I have a program that sends a series of set requests to an SNMP > > device. All the set-requests work perfectly in standalone. When executed > one > > after another, they execute successfully 5 out of six times in the other > > instance I get a Timeout exception in a Random place. > > > > When I attempted to trace the packets with Wireshark (assuming that it is > a > > device problem) I noticed that the response for the set request was > properly > > received by the machine immediately, but It was not captured by the > > particular Westhawk session. > > > > I am not sure how to fine tune the WestHawk API to prevent it from > dropping > > any packets. Thanks, Kevin > > > > -- > -- Birgit Arkesteijn, birgit@westhawk.co.uk, > -- Westhawk Ltd, Albion Wharf, 19 Albion Street, Manchester M1 5LN, UK > -- Company no: 1769350 > -- Registered Office: > -- 15 London Road, Stockton Heath, Warrington WA4 6SJ. UK. > -- tel.: +44 (0)161 237 0660 > -- > _______________________________________________ > snmp mailing list > snmp@snmp.westhawk.co.uk > http://snmp.westhawk.co.uk/mailman/listinfo/snmp > > > _______________________________________________ > snmp mailing list > snmp@snmp.westhawk.co.uk > http://snmp.westhawk.co.uk/mailman/listinfo/snmp > From birgit at westhawk.co.uk Thu Jun 5 12:09:40 2008 From: birgit at westhawk.co.uk (Birgit Arkesteijn) Date: Thu Jun 5 11:09:47 2008 Subject: [snmp] SNMP Set Response is lost / packet dropped by Westhawk API In-Reply-To: <002301c8c671$d7bc8700$12fd5980@7C911> References: <002301c8c671$d7bc8700$12fd5980@7C911> Message-ID: <4847BB64.1030300@westhawk.co.uk> Hi Josh, On 04/06/08 19:35, Josh Bers wrote: > Birgit, > > In trying to understand why the stack might drop some responses, there are a > few questions: > > 1. Are set requests retried the same way as get requests? Namely if a > response is not received they are resent? (this might be risky for setting > things that should only be set once...) Yes, they are. The context that handles the retries is unaware of the content that it sends. > 2. If an observer for receiving PDU responses does a lot of processing, what > happens to the receiver thread that is listening on the inbound socket? > > If answer to above is that the receiver thread and the thread that calls > update on the observer are the same then there may be a problem of dropping > UDP packets... > Since there are no guarantees on UDP and there is a finite queue that the OS > has for UDP sockets (I think 5 deep is the default), then the TCP/IP in the > OS may well just drop response PDU's when there are a lot of them and the > Observers are causing calls to receivePacket to be spaced out. The receiver thread and the thread that calls update on the observer are the same. This does mean indeed that packets might be dropped if they come in bigger numbers and/or the processing of the response takes longer. BTW, in contrast, the ListeningContext fires a thread for every incoming message. > 3. If the answer to the above is that there is a separate thread to listen > for response on the socket and a separate one that dispatches to observers, > then there may be a queuing problem. How does the stack queue up these > received packets? This is not the case. A queuing system (FIFO) might be an option to add to the stack, but it does mean doubling the number of threads, which will cause other problems. > Josh It's a bit of a balancing act, but at least with the current solution the user (programming the stack) has the option to add additional threads, whereas with a queueing system, they are forced upon her/him. Cheers, Birgit -- -- Birgit Arkesteijn, birgit@westhawk.co.uk, -- Westhawk Ltd, Albion Wharf, 19 Albion Street, Manchester M1 5LN, UK -- Company no: 1769350 -- Registered Office: -- 15 London Road, Stockton Heath, Warrington WA4 6SJ. UK. -- tel.: +44 (0)161 237 0660 -- From birgit at westhawk.co.uk Thu Jun 5 12:14:59 2008 From: birgit at westhawk.co.uk (Birgit Arkesteijn) Date: Thu Jun 5 11:15:07 2008 Subject: [snmp] SNMP Set Response is lost / packet dropped by Westhawk API In-Reply-To: <81367b5d0806041209u34ec5533uff199c9d976c5880@mail.gmail.com> References: <48451243.2040800@westhawk.co.uk> <002301c8c671$d7bc8700$12fd5980@7C911> <81367b5d0806041209u34ec5533uff199c9d976c5880@mail.gmail.com> Message-ID: <4847BCA3.2030201@westhawk.co.uk> Hi Kevin, Not a problem. Reusing a context (where possible) is always a good idea. Also, when calling destroy() immediately, you run the risk that the context is gone before the response is received. We always advocate sending a next (or sequential) request when the response to the previous request is received (in update()). This way you allow the network to drive the speed of your requests. Cheers, Birgit On 04/06/08 20:09, Kevin Bond wrote: > Hi Birgit, > > Thanks for your quick response and I apologize for my delayed reply. > > I resolved the issue after making some changes to the frequency of > SNMP V2C Context creation. I have given root cause observed to help you > get a better handle on the issue. > > In the project, I had an SNMP Adapter layer where I used to create a > Context to execute the command and destroy the context immediately after > the execution. This layer had no visibility on the frequency of execution. > > In the particular scenario, the layer using the SNMP Adapter sent a > lot of set requests in sequence (one by one). In this scenario WestHawk > stack had the mentioned problem in receiving the response from the > device. I changed the context handling to use the same SNMP V2c Context > for subsequent requests to avoid the problem. > > I am using the latest 6.0 stack and only V2C Contexts. > > Hope this helps, > Kevin -- -- Birgit Arkesteijn, birgit@westhawk.co.uk, -- Westhawk Ltd, Albion Wharf, 19 Albion Street, Manchester M1 5LN, UK -- Company no: 1769350 -- Registered Office: -- 15 London Road, Stockton Heath, Warrington WA4 6SJ. UK. -- tel.: +44 (0)161 237 0660 -- From jbers at bbn.com Wed Jun 11 11:02:12 2008 From: jbers at bbn.com (Josh Bers) Date: Wed Jun 11 15:02:45 2008 Subject: [snmp] V3 Discovery problem Message-ID: <000001c8cbcb$c5788a30$54fd5980@7C911> I have run into a (new) SNMP V3 discovery problem with the stack (version 6.0 most recent release version): The problem shows up when I try to contact an agent that is misconfigured and then re-configured correctly. During this time the stack is running, attempting to request data from the agent. 1. The agent has V3 enabled, however, the username that I am using is not configured correctly on the agent. It appears that discovery half succeeds, 2. I reconfigure the agent to know about the username and restart it, 3. The stack (which has not been restarted) then gets the following exception while doing discovery: java.lang.NullPointerException" java.lang.NullPointerException at java.util.Hashtable.containsKey(Hashtable.java:314) at uk.co.westhawk.snmp.stack.TimeWindow.isTimeLineKnown(TimeWindow.java:218) at uk.co.westhawk.snmp.beans.UsmDiscoveryBean.discoveryTimeLine(UsmDiscoveryBea n.java:299) at uk.co.westhawk.snmp.beans.UsmDiscoveryBean.startDiscovery(UsmDiscoveryBean.j ava:195) at uk.co.westhawk.snmp.stack.SnmpContextv3Basis.discoverIfNeeded(SnmpContextv3B asis.java:953) at uk.co.westhawk.snmp.stack.SnmpContextv3Basis.addPdu(SnmpContextv3Basis.java: 562) at uk.co.westhawk.snmp.stack.SnmpContextv3Basis.addPdu(SnmpContextv3Basis.java: 478) at uk.co.westhawk.snmp.stack.SnmpContextv3Pool.addPdu(SnmpContextv3Pool.java:39 1) at uk.co.westhawk.snmp.stack.Pdu.send(Pdu.java:221) at uk.co.westhawk.snmp.stack.Pdu.send(Pdu.java:204) Any help with fixing/diagnosing this problem would be much appreciated. Josh Josh Bers Senior Engineer, Mobile Networking Systems BBN Technologies web: www.bbn.com ph: (617) 873-4262 fax: (617) 873-4523 From birgit at westhawk.co.uk Thu Jun 12 16:06:54 2008 From: birgit at westhawk.co.uk (Birgit Arkesteijn) Date: Thu Jun 12 15:07:04 2008 Subject: [snmp] Re: V3 Discovery problem In-Reply-To: <000001c8cbcb$c5788a30$54fd5980@7C911> Message-ID: <675825.123411213279614139.JavaMail.root@zimbra> Hi Josh, The java.lang.NullPointerException happens on the line engineLookup.containsKey(snmpEngineId); so snmpEngineId is null. If snmpEngineId is null this *should* mean that discoveryEngineId() is called. However, if discoveryEngineId() is called, this *should* mean that snmpEngineId is discovered. It sounds a bit like chicken and egg. If you set debug to 5, it will print everything to do with discovery and setting the TimeWindow. Could you please do this and post back the results? Cheers, Birgit ----- "Josh Bers" wrote: > I have run into a (new) SNMP V3 discovery problem with the stack > (version 6.0 most recent release version): > > > > The problem shows up when I try to contact an agent that is > misconfigured and then re-configured correctly. During this time the > stack is running, attempting to request data from the agent. > > > > 1. The agent has V3 enabled, however, the username that I am using > is not configured correctly on the agent. It appears that discovery > half succeeds, > 2. I reconfigure the agent to know about the username and restart > it, > 3. The stack (which has not been restarted) then gets the > following exception while doing discovery: > > > > > java.lang.NullPointerException" > java.lang.NullPointerException > at java.util.Hashtable.containsKey(Hashtable.java:314) > at > uk.co.westhawk.snmp.stack.TimeWindow.isTimeLineKnown(TimeWindow.java:218) > at > uk.co.westhawk.snmp.beans.UsmDiscoveryBean.discoveryTimeLine(UsmDiscoveryBean.java:299) > at > uk.co.westhawk.snmp.beans.UsmDiscoveryBean.startDiscovery(UsmDiscoveryBean.java:195) > at > uk.co.westhawk.snmp.stack.SnmpContextv3Basis.discoverIfNeeded(SnmpContextv3Basis.java:953) > at > uk.co.westhawk.snmp.stack.SnmpContextv3Basis.addPdu(SnmpContextv3Basis.java:562) > at > uk.co.westhawk.snmp.stack.SnmpContextv3Basis.addPdu(SnmpContextv3Basis.java:478) > at > uk.co.westhawk.snmp.stack.SnmpContextv3Pool.addPdu(SnmpContextv3Pool.java:391) > at uk.co.westhawk.snmp.stack.Pdu.send(Pdu.java:221) > at uk.co.westhawk.snmp.stack.Pdu.send(Pdu.java:204) > > > > Any help with fixing/diagnosing this problem would be much > appreciated. > > > > Josh > > > > Josh Bers > > Senior Engineer, Mobile Networking Systems > > BBN Technologies > > web: www.bbn.com ph: (617) 873-4262 fax: (617) 873-4523