Au Aix Packet Dropping PDF

Copyright IBM Corporation 2013 TrademarksSimulating dropped TCP/IP packets on IBM AIX Page 1 of 26

Simulating dropped TCP/IP packets on IBM AIX

Jerry Stevens ([email protected])Software Developer, WebSphere Service Registryand Repository Development teamIBM

Skill Level: Intermediate

Date: 15 Mar 2013

Commercial software products are often developed and tested on local high-speed, high-capacity networks, where there is little congestion or packet loss.The network conditions in deployment environments may not be so ideal. Heavycongestion or hardware errors on networking equipment might result in lostpackets, which can severely affect product performance. This article shows youhow to use and customize a provided facility for simulating dropped TransmissionControl Protocol/Internet Protocol (TCP/IP) packets on IBM AIX to allowyou to anticipate how well a product will behave under less-than-ideal networkconditions.

IntroductionThis article presents:

An AIX kernel extension to permit a specified percentage of TCP/IP packets toand from a designated host to be dropped randomly so as to simulate adversenetwork conditions.

A utility to load, activate, and unload the kernel extension. C and Java utilities to monitor total packet throughput to the target host and

actual packets dropped. Full source for the kernel extension with C and Java utilities so that the software

can be built, customized, and developed according to local requirements.

All the components needed to build and run this packet dropping simulator areprovided with this article as a downloadable compressed file. The How to build anduse the kernel extension as supplied section enables you to get started and use it assoon as possible.

The kernel extension is designed to be activated on a single source node from whichit will selectively drop packets either outbound to a target node or received from it. It

developerWorks ibm.com/developerWorks/

Simulating dropped TCP/IP packets on IBM AIX Page 2 of 26

is not necessary to install the packet drop simulator on the target node. The targetnode itself does not even need to be AIX-based but just needs to support the TCP/IPprotocol.

The article first presents some background to TCP/IP performance and why droppedpackets can have a serious impact on application performance. You will then see howto build and use the kernel extension as provided. This enables you to get up andrunning quickly with it.

The supplied functionality is deliberately very simple. It allows you to set a givenpercentage of TCP/IP packets to be dropped randomly to and from a specifiedhost, with all other hosts and protocols unaffected and no retained history. Becausein practice, you might need to add further functionality into the kernel extensionaccording to your own requirements, you will also be shown details as to how theextension works and how it can be customised. You can then decide to do this if youneed to:

Permit packet dropping to and from multiple hosts. Support different packet drop rates for each of those hosts. Create historic logs of total packets received, transmitted, and dropped in both

directions. Implement the packet drop simulator on different versions of AIX or C compilers

to those used when the software was written.

The kernel extension could be readily developed further to simulate other networkissues that could give rise to performance problems, such as packet corruption,packets arriving out of order, and jitter.

It is recommended the kernel extension is only used on non-productionsystems.

Background

First let's review some basic terms that are important when considering networkperformance, bandwidth, latency, and congestion.

Bandwidth is the capacity of the network, that is the rate at which data can be pushedacross it. This is usually measured in Megabits per second (Mbps), where oneMegabit is 10^6 bits. It can be thought of as the width of the network pipeline.

Latency is the time it takes for one piece of data to traverse the network. This isusually measured in milliseconds. It can be thought of as the length of the networkpipeline.

The difference between bandwidth and latency is illustrated in Figure 1.

ibm.com/developerWorks/ developerWorks


Figure 1. Latency and bandwidth and the network pipeline

Congestion is the effect of heavy usage of the network. It can result in delaysin propagating data, loss of packets resulting in packet retransmission, and aninability to make connections to the network. Packet loss is usually measured as apercentage. TCP can normally handle losses up to 0.1% with little direct impact. If thelosses increase above this, the effect can be more severe.

Network latency, bandwidth, and congestion can have a serious impact onapplication performance. Consider, for example, the situation where a front-endapplication on one system makes frequent calls across the network to anothersystem which hosts a database used by the application. The traffic across thenetwork to the database might consist of many bursts of small data, or might consistof fewer sets of relatively large data transfers. Either way, the ability of the network totransfer this data reliably and speedily can be a significant factor in the application'sperceived end-to-end performance by a customer.

Application development teams often have access to isolated, high-speed networksthat can be much more lightly used than the environment into which an applicationis to be deployed. This can mean the effect of poor network performance on theapplication may never be observed within the environment in which it is developedand tested. This can, in turn, mean that the performance observed within acustomer's environment might be very different to that observed in the environmentfrom which it is tested.

This article focuses on the effect of dropped packets on a particular application'sperformance. Due to the fact that there can be very high numbers of packetstransferred between a front-end and a back-end system and the patterns in whichthese application packets are transmitted are very specific to the nature of the front-end and back-end applications, the simplest approach to understand the impact onthe application is to simulate the dropping of packets.

Packets can be dropped when transferring data between systems for two keyreasons:

Heavy network utilization and resulting congestion



Faulty network hardware or connectors

TCP is designed to be able to react when packets are dropped on a network. Whena packet is successfully delivered to its destination, the destination system sendsan acknowledgement message back to the source system. If this acknowledgementis not received within a certain interval, that may be either because the destinationsystem never received the packet or because the packet containing theacknowledgement was itself lost. In either case, if the acknowledgement is notreceived by the source system in the given time, the source system assumes that thedestination system never received the message and retransmits it. It is easy to seethat if the performance of the network is poor, packets are lost in the first place, andthe increased load from these retransmit messages is only increasing the load on thenetwork further, meaning that more packets will be lost. This behaviour can result invery quickly creating a critical situation on the network.

There are some freely available sophisticated software products that can help tosimulate different characteristics of varying network performance. These include:

WANem: Runs from Knoppix Linux based CD ipfilter: Public domain software but not fully ported to AIX dummynet (FreeBSD) ALTQ (FreeBSD)

WANem runs from a Knoppix-based CD and can simulate a large range of differentnetwork characteristics. It does offer the ability to drop a random number of packetswith filtering options but would need to run on a separate system in between thesource and target system.This may not be convenient, for example if the source andtarget systems are installed on dedicated high-speed networks.Building the kernelextension and utilities

Although ipfilter looked very promising, it had only been partially ported to AIX 5.3and it became evident that it would be quicker to write a bespoke utility providing theminimal functionality required rather than porting the whole of ipfilter to both AIX 6.1and 7.1.

dummynet is a component of the FreeBSD operating system and has alsobeen ported to other platforms including Linux and Microsoft Windows. Thedisadvantage of using this though is that to simulate packet drops for packetsoriginating from an AIX host, an intermediate FreeBSD/Linux host has to beconfigured and set up with dummynet.

ALTQ is an alternate queueing framework that helps to provide bandwidth control,mostly on BSD routers.

There are also other packages commercially available such as LANforge ICE butthese were ruled out for cost reasons. LANforge also requires dedicated hardware.



How to build and use the kernel extension as suppliedRequirementsThe packet drop simulator was built and tested in the following environments:

AIX V7.1.4.0 with gcc V4.7.2-1 AIX V6.1.2.0 with gcc V4.2.0 AIX V6.1.4.0 with gcc V4.2.0

The kernel extension and associated C programs were compiled using the gcccompiler. It should be straightforward to use an AIX C compiler instead if required,although some of the compilation flags will need to be changed.

Note that it will be necessary to install the AIX bos.adt.syscalls file set to build thekernel extension. This will be available on your AIX installation media. The file set willnot be required on systems were the kernel extension is only to be deployed as longas the AIX level is identical to the build system. If there are any differences in AIXlevels between the two systems, the kernel extension should be built on each.

The kernel extension, the kctrl utility to load and activate it and the C example controland monitor are all built with the AIX make facility. A makefile to build all of thesecomponents is included in the downloadable zip file.

PrecautionsTwo important factors should be kept in mind before using this utility:

1) Care should be exercised when developing and testing any kernel extension. Itis common for errors to result in system failure. A dedicated test system is ideal,but avoid using systems that are also used for other purposes or by other users. Ifthe system is located in a remote data centre, you should also ensure you have themeans for it to be rebooted when required if it does not reboot automatically.

2) Avoid the use of flood pings to test the utility on a shared network as this mayseverely impact other users.

Building the kernel extension and utilitiesFirst make sure you have met the requirements on your development system asoutlined above.

Next, extract the downloadable attachment into a working directory of your choice onthe target AIX system.

The download does not include pre-built binaries so as to avoid the risk of runningit on incompatible operating system levels. The software will therefore need to bebuilt once it has been extracted. To this, change directory to the working directory youcreated above and then run the make commands as follows:



# make gcc -Wall -maix64 -ffreestanding -msoft-float -o pdrop_kernex64.o -c pdrop_kernex.c ld -b64 -o pdrop_kernex64 pdrop_kernex64.o -e pdrop_init -bI: /home/jerry/kernext/kernex.exp -bI:/home /jerry/kernext/netinet.exp -bE:/home/jerry/kernext/pdrop_syscall.exp -lsys -lcsys rm -f pdrop_kernex ar -X64 -r -v pdrop_kernex pdrop_kernex64 ar: Creating an archive file pdrop_kernex. a - pdrop_kernex64 gcc -Wall -maix32 -o pdrop_ccm32 .c -Xlinker -bI: /home/jerry/kernext/pdrop_syscall.exp gcc -Wall -maix64 -o pdrop_ccm64 pdrop_ccm.c -Xlinker -bI: /home/jerry/kernext/pdrop_syscall.exp gcc -maix64 -o kctrl kctrl.c gcc -Wall -I/usr/java6/include -I. -shared -fPIC pdrop_jni.c -o libpdrop_jni.so -Xlinker -bI:/home/jerry/kernext/pdrop_syscall.expTarget "all" is up to date.#

After the make command is run successfully, all C components of the packet droputility required to use it will be ready for use.

Note that the Java control and monitor application need to be built separately asdescribed in the Control and monitor applications section

Loading and activating the kernel extension

One of the binaries that will have been build in the proceeding step is the kctrlprogram. This permits the kernel extension to be loaded, activated, and unloaded. Toload and activate the kernel extension, proceed as follows:

# ./kctrl /home/jerry/kernext/pdrop_kernex64

Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)ndlExtension Successfully loaded, kmid is 1353523200

Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)ndi Extension Initialized

Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)nde#

At this point, the kernel extension will be loaded and active, but will not drop anypackets to a target host until either the C or Java control and monitor utilities areused.

Initiating packet dropping and monitoring

Having loaded and initialized the kernel extension, you can now see how to use thesupplied C utility, pdrop_ccm, to control and monitor the setting of packets to a targethost.



pdrop_ccm is invoked with arguments that specify the location where the kernelextension was built, the target host name, and the drop rate.

# ./pdrop_ccm64 /home/jerry/kernext/pdrop_kernex64Usage: ./pdrop_ccm64 #

You can use this, for example, with the host name fred and a drop rate of 0.5% asfollows:

# ./pdrop_ccm64 /home/jerry/kernext/pdrop_kernex64 fred 0.5Kernel extenstion is loaded.Setting drop ip to: x.y.z.lDrop rate: 0.50.

Total In In dropped PC in dropped Total Out Out dropped PC out dropped 0 0 0.00 0 0 0.0010120 57 0.56 10174 53 0.5222224 134 0.60 22340 116 0.52#

In this example, shortly after we started pdrop_ccm we started sending IP traffic to thetarget host. The pdrop_ccm utility will keep running until it is interrupted with a ctrl+cand will print an update line every 10 seconds.

Please note that the first command line parameter, the kernel extension name, mustbe specified exactly as it was when loaded with kctrl.

The drop percentage argument is optional. If it is not supplied, pdrop_ccm will displaythe current drop rate only and will not re-set it.

A Java-based control and monitor utility is also provided and this is described in theControl and monitor applications section.

That is all there is to get started with the kernel extension.

The rest of this article presents more details on how the kernel extension works andhow it and the control and monitor applications can be developed and customized forindividual use.

How the kernel extension works

The AIX kernel extension is written in the C programming language. The kernelfunction exposes access to it through a set of system calls. The C and Java controland monitor applications utilize these system calls to drive the kernel extensionto drop packets and to and from a particular host at a given rate. For the Javaenvironment, a JNI wrapper class provides access to the kernel extension's systemcalls from Java.

The kernel extension itself consists of:



A set of system calls which control the dropping of packets and the collection ofpacket statistics

Two functions which hook into the kernel's networking layers. One of these is forinbound traffic and the other for outbound. These functions are called whenevera packet is received or is to be dispatched to control the dropping of packets.

The system calls are accessed by the user-level code to set the remote systemto which incoming and outgoing packets should be dropped and what percent ofpackets should be dropped. These system calls also contain functions to retrievecounters defining how many packets have been dropped in both inbound andoutbound directions and also a function to reset the counters.

For the C environment, these system calls are used directly in the example controland monitor application pdrop_ccm.c, which we used in the Initiating packet droppingand monitoring section above. For the Java environment, the system calls can beused using JNI. The JNI wrapper can then be used by a bespoke Java applicationto control and monitor the dropping of packets. Again, a sample Java application isprovided as a downloaded resource with this article. This is called pdrop_jcm.java.Refer to the Java control and monitor section for further details on the example Javautility.

Figure 2 illustrates the different components to the kernel extension and the flow ofdata through it. The following two sections discusses the flow of packet flow throughthe extension, both outbound data to be sent to the target node and inbound datareceived from it.

Outbound traffic flow

First, consider the case of outbound traffic, where an application under test submitsdata to be transmitted across the network. Refer to point A in the Figure 2. Theapplication submits a set of data to the kernel for sending to the remote node onthe network. The TCP subsystem breaks this down into a number of packets whichare normally then sent to the appropriate network device driver for dispatch acrossthe network. When using the kernel extension, however, the hook outbound_fw isset in the IP layer which results in the kernel extension's outbound filter functionpdrop_outbound_filter being called for every packet being dispatched. Thisretrieves the IP header from the packet and matches it against the target nodefor which packets are to be dropped. If the node does not match, the function,ip_output_post_fw is called which routes the packet to the appropriate interface onthe host. If the node does match, then a random decision whether to pass the packetis made, which is weighted according to the number of packets to be dropped. Ifthis decision is that the packet should be sent, the ip_output_post_fw function iscalled and the packet is passed. Otherwise, the mbuf holding the data is freed andthe outbound filter returns and the packet is not sent.



Inbound traffic flow

For inbound traffic, a corresponding operation applies when data is received fromthe network by the device driver. Refer to point B in Figure 2. The device driverforwards the packet of data to the TCP/IP subsystem. This now calls the inboundfilter which again has been set by the kernel extension by assigning the inboundhook inbound_fw to the pdrop_inbound_filter function. This function makes thedecision whether to pass the packet or not. If the packet is not from the targetnode for which packets are being dropped, it is automatically passed by callingthe ipintr_noqueue_post_fw function. If it is from the target node, then again arandom decision is made whether to pass the packet according to the requiredpercentage of packets that are to be dropped. If the decision is to send the packet,the ipintr_noqueue_post_fw function is called and the packet is passed. Otherwise,the mbuf holding the data is freed and the inbound filter returns and the packet is notpassed up to the higher layers.

The AIX IP filtering hook and kernel services described above are explained in theIBM AIX 7.1 information center.



Figure 2. The packet drop kernel extension



Inbound filterThis is the inbound filter function:

/*** This function is the filter for incoming traffic*/void pdrop_inbound_filter(ifp, m, args)struct ifnet *ifp;struct mbuf *m;inbound_fw_args_t *args;{ struct ip *ip_in;

if (PDROP_TRUE == amDropping) { ip_in = mtod(m, struct ip *);

if ( (ip_in->ip_p == IPPROTO_TCP) && (dropip.s_addr == ip_in->ip_src.s_addr)) { fetch_and_addlp((atomic_l) &total_in, 1); if (dropMod != 0) { if ( (pdrop_random() % dropMod) == 1) { fetch_and_addlp((atomic_l) &nin_dropped, 1); if (m != NULL) { m_freem(m); } return; } } }}ipintr_noqueue_post_fw(ifp, m, args);return;}

The arguments to the filter function include an mbuf containing the received data.The mtod macro is used to convert the pointer to the mbuf into a pointer to thereceived data, which is addressed as an IP header. A counter of the total number ofreceived packets from the target address is maintained.

The function then compares the IP address selected for dropping to the sourceaddress specified in the received packet. If these match, then the pdrop_random()function is called, which uses the modulus function to select the requiredpercentage of packets for dropping. If the packet is to be dropped, then an internalcounter of dropped packets is incremented. If it is not to be dropped, then theipint_noqueue_post_fw function is called to pass the packet to TCP for processing.

The fetch_and_addlp system functions are called to increment the counts of thetotal number of received and dropped packets. These ensure that the counts aremaintained atomically so as to be thread safe.

You can see from this example that it is possible to specify which protocol is tofiltered. In this case, we are only dropping packets for the TCP protocol. For testing,



it is a good idea to make it use another protocol, such as Internet Control MessageProtocol (ICMP) so then the packet dropper can be used with commands, such asping. To do that, the IPPROTO_TCP constant in the test above would be changed toIPPROTO_ICMP.

Note: amDropping, dropip, and dropMod in the above extract are defined as globalvariables.

Outbound filter

This is the outbound filter function:

/*** This function is the filter for outbound network traffic*/int pdrop_outbound_filter(ifp, m, args)struct ifnet *ifp;struct mbuf *m;outbound_fw_args_t *args;{ struct ip *ip_out;

if (PDROP_TRUE==amDropping) { ip_out = mtod(m, struct ip *);

if ( (ip_out->ip_p == IPPROTO_TCP) && (dropip.s_addr == ip_out->ip_dst.s_addr)) { fetch_and_addlp((atomic_l) &total_out, 1); if (dropMod != 0) { if ( (pdrop_random() % dropMod ) == 1) { fetch_and_addlp((atomic_l) &nout_dropped, 1); if (m != NULL) { m_freem(m); } return 0; } } }}

ip_output_post_fw(ifp, m, args);return 0;}

This works very similar to the inbound filter. Counters are maintained for the totalnumber of packets destined for the target IP address and for the total number ofoutbound packets dropped.

The pointer to the mbuf is freed if the packet is to be dropped.

Note: amDropping, dropip, and dropMod in the above extract are defined as globalvariables.



Randomizing the packets to be dropped

It is important for the kernel extension to select packets at random for dropping. Ifsay, packets were dropped after a set number of packets had been sent, this wouldnot necessarily be a true emulation of what happens on a heavily used network.Further, it is possible that the software under test may behave differently when itmisses packets at regular and irregular intervals.

The standard C library rand() call to generate random numbers cannot be used ina kernel extension. This is because such functions are not safe to use in a the re-entrant kernel environment. If you attempt to use this function, the system might failwhen it is called from the kernel extension.

A simple function to mimic a call to random was therefore used. This generates along random number based on the following inputs:

The seconds field from the current time The nano-seconds field from the current time The number of calls to the random function The number of processor ticks since system boot

This is the source of the random function that is used in the kernel extension.

long pdrop_random(){ static long calls = 0; struct timestruc_t ts; long x = 0;

curtime(&ts);

// lbolt is the number of clock ticks since boot, see HZ in /usr/include/sys /m_param.h for number of ticks/sec

fetch_and_addlp((atomic_l) &x, (long) ts.tv_nsec + (long) ts.tv_sec + (long) lbolt + calls++);

return x;}

This is not of course a true random function but is a good enough mechanism forassuring that packets are dropped in a sufficiently irregular manner. As the callsvariable is static, it is incremented atomically with the fetch_and_addlp kernelservice.

System calls provided in the packet drop kernel extension

In this section, you are shown details of the various system calls provided in thekernel extension. These will be of interest if you need to write your own control andmonitor applications or customize the kernel extension.



Setting the target address: int pdrop_set_drop_address(struct in_addr*block_addr)

This function takes a pointer to an in_addr struct as an argument. The method maybe called from the application level from a string-based host name as follows:

int setDropAddress(char *hostname){ int status = 0; struct in_addr block_addr;

if (1 == inet_aton(hostname, &block_addr)) { if (PDROP_TRUE == pdrop_set_drop_address(&block_addr)) status = 1; } return status;}

Retrieving the target address: int pdrop_get_drop_address()

This function returns the target address as a 32-bit unsigned integer.

Setting the rate at which packets are dropped: extern voidpdrop_setDropMod(long m);

This function takes a long input value, which represents how often packets should bedropped. The way this is worked out is that the kernel extension generates a randomnumber and then takes a modulus of that random number with the long value. If theresult is one, then the packet is dropped, otherwise it is passed.

The call is designed in this way as it is not possible to perform floating pointoperations within the kernel extension.

Here is a simple way of calling pdop_setDropMod() from the application level basedon a double percentage value:

void setDropPC(double d){ long x = 0; x = 100.0 /d; pdrop_setDropMod(x);}

So, for example, if it was required to drop 0.1% of packets, then d would be 0.1 andthe long value passed to pdrop_setDropMod would be 1000.

Retrieving the rate at which packets are dropped: extern longpdrop_getDropMod();

This function returns the long representation of how often packets should bedropped, which is expressed in the same way as a call to pdrop_setDropMod();



Activate the dropping of packets: void pdrop_startDropping()

When the kernel extension is started, packets are not dropped until thepdrop_startDropping() function is called. You should call this function after setting thetarget address and the rate at which packets should be dropped.

Stop the dropping of packets: void pdrop_stopDropping()

This function stops the dropping of packets. The functions pdrop_startDropping()and pdrop_stopDropping() can be called as required to enable and disable packetloss.

Query whether packets are being dropped: int pdrop_amDropping()

This function returns a 1 if the kernel extension is currently dropping packets and a 0it is not.

Retrieving kernel extension counters: void pdrop_getstats(struct pdrop_stats*p)

This function returns a structure which contains details of:

The total number of packets received from the target address The total number of packets received from the target address which were

dropped The total number of packets sent to the target address The total number of packets sent to the target address which were dropped.

The definition of the structure used is in the header file, kernext_pdrop.h.

Here is an example of how the counters might be received from the application level:

pdrop_stats_t j;pdrop_getstats(&j);

printf("Total in: %d.\n", j.total_in);printf("In dropped: %d.\n", j.nin_dropped;);printf("Total out: %d.\n", j.total_out);printf("Out dropped: %d.\n", j.nout_dropped);

Resetting the internal kernel extension counters: void pdrop_reset_counters()

This function is used to reset the counters returned by pdrop_getstats to zero.

Configuring the kernel extension

The How to build and use the kernel extension as supplied section providedinformation about how to build the kernel extension and its utilities. This sectionprovides:



More details on the build process Information on AIX 6.1 and AIX 7.1 build issues Details about exporting the system calls provided in the kernel extension to the

application level Details about loading and activating the kernel extension Information about the use of system logging to record when the extension is

loaded and unloaded

AIX 6.1 and AIX 7.1 build issuesBeginning with AIX 6.1, the AIX operating system simplified its kernel environmentby providing only the 64-bit kernel. AIX 6.1 and AIX 7.1 maintain application binarycompatibility with previous AIX versions as specified above, but device drivers andkernel extensions that are only 32-bit cannot be built on AIX 6.1 or AIX 7.1.

As this article presents the kernel extension as suitable for AIX 6.1 and AIX 7.1, it hasbeen built in the 64-bit mode.

The build processA Makefile is included with the compressed file provided with this article. This can beused to build the kernel extension and application layer programs associated with it.Note that this Makefile has been written for use with the AIX make utility and it willneed to be modified if you wish to use gnu make.

The kernel extension itself is built with the following commands:

gcc -maix64 -ffreestanding -msoft-float -o pdrop_kernex64.o -c pdrop_kernex.c

ld -b64 -o pdrop_kernex64 pdrop_kernex64.o -e pdrop_init -bI:/usr/lib/kernex.exp -bI:/usr/lib/netinet.exp -bE:/home/jerry/kernext/pdrop_syscall.exp -lsys -lcsys

As noted earlier, the kernel extension is built as a 64-bit binary. The -ffreestandingand -msoft-float options are used to prevent the use of floating point instructionsto manipulate certain data structures. This was required on AIX 7.1, but was notnecessary on AIX 6.1.

The ld command specifies the entry point into the kernel extension, which in thiscase is the pdrop_init() function. This function is called when the kernel extension isactivated.

The ld command also refers to the files kernex.exp and netinet.exp. These files areprovided with the bos.adt.syscalls file set. This file set may not be installed on yourtest system in which case you will need to ask the system administrator to install it.

Note that it is necessary to make one small modification to the /usr/include/sys/socketvar.h header file for gcc to compile the kernel extension satisfactorily. Youshould retain a safe copy of the file and locate the following line:

extern struct free_sock_hash_bucket free_sock_hash_table[];



This should be changed to:

extern struct free_sock_hash_bucket * free_sock_hash_table;

The kernel extension makes this #define in the source:

#define _MSGQSUPPORT 1

This is necessary to prevent the fd_select() system call being made which is notavailable to the kernel extension. Using #define changes the fd_select() call to theoriginal select() call, which is available to the kernel extension.

Exporting the provided system callsThe pdrop_syscall.exp file defines the kernel extension system calls that areexported to the application level. The contents of this file are as follows:

#!/unixpdrop_set_drop_address syscall3264pdrop_get_drop_address syscall3264pdrop_setDropMod syscall3264pdrop_getDropMod syscall3264pdrop_startDropping syscall3264pdrop_stopDropping syscall3264pdrop_amDropping syscall3264pdrop_getstats syscall3264pdrop_reset_counters syscall3264

If you need to customize the kernel extension to include further system calls, youwill need to amend this file. The syscall3264 identifier at the end of each line makesthe system calls available to both 32-bit and 64-bit processes. The flag may also beset to syscall32 to support calls from 32-bit processes only or syscall64 for 64-bitprocesses only. If the flag is not set for the correct target process environment, theprocess will fail with a segmentation fault when the system call is made. For furtherdetails on how to set this identifier refer to the topic, Exporting Kernel Services andSystem Calls.

Use of kctrl to control loading of the kernel extensionThe kernel extension is loaded and unloaded with the kctrl program provided asdescribed earlier. The program should be invoked with the full path name of thekernel extension. It then interactively accepts the following commands:

q checks whether the kernel extension has been loaded l loads the kernel extenstion I initializes the kernel extension t terminates the kernel extension u unloads the kernel exension e exits the utility

Here is an example where kctrl is used to query, load, initialize, terminate, unload,and quit the utility:



# ./kctrl /home/jerry/kernext/pdrop_kernex

Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)ndq Extension is not loaded

Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)ndlExtension Successfully loaded, kmid is 1353052160

Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)ndi Extension Initialized

Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)ndq Extension is loaded, with kmid 1353052160

Enter choice, (l)oad, (u)nload, (i)nit, (t)erm, (q)uery or (e)nde#

You can also check whether the kernel extension has been loaded by using the AIXgenkex command, for example:

# genkex | grep -i kernf1000000c02be000 2000 /home/jerry/kernext/kernext_hello#

In this example, the kctrl executable works with interactive user input. It would bestraightforward to modify kctrl to take the command as another argument on thecommand line and thereby make it easy to incorporate within system startup andshutdown scripts. However, this is not generally recommended given the nature ofthe utility.

Logging in the kernel extension

The kernel extension uses the syslogd daemon to record when the extension isloaded and unloaded. This can be useful for debugging or auditing purposes. Toenable this logging:

Ensure that logging is enabled in /etc/syslog.conf. For example, the followingline can be appended to this file:

*.debug /var/log/syslog.out rotate size 100k files 4

Ensure that the log file enabled in /etc/syslog.conf exists. Refresh the syslogd subsystem (refresh -s syslogd)

Use with other IP protocols

Although the extension is primarily considered for TCP/IP, it would be simple tochange it to work with other IP-based network protocols. This was discussed in theHow the kernel extension works section.



Control and monitor applicationsTwo example applications that can use the system calls exposed by the kernelextension are provided. One is for the C environment and the other for the Javaenvironment.

These applications show you how to write your own custom applications to controlpacket dropping. You can decide to do this if you need to write your own automatedtest framework to simulate different network conditions.

It should be noted that only the applications that collect statistics directly fromthe kernel extension can provide meaningful statistics on packets that have beendropped. Operating system provided utilities that report on dropped packets will notinclude details of packets that have been dropped through the kernel extension. Thisis because the packets are dropped by the extension before they are passed throughthe TCP/IP subsystem for dispatch or delivery.

C control and monitorThe test application, pdrop_ccm.c, is provided in the downloads section of this article.This is invoked with a target host name and drop percentage on the command line.The application passes details of the target host and the drop rate to the kernelextension and then monitors the inbound and outbound packets to the target systemevery 10 seconds.

The packet drop system calls can be invoked from both 32-bit and 64-bit applications.The makefile provided demonstrates this by building both 32-bit and 64-bitversions of the control and monitor application. These are called pdrop_ccm32 andpdrop_ccm64 respectively.

pdrop_ccm shows the total number of input and output packets, the number of inputand output packets that were dropped, and the numbers of dropped packets as apercentage. It takes the name of the kernel extension, the name of the target hostas arguments and an optional value specifying the target drop rate. If this thirdparameter is not specified on the command line, no change to the drop rate is made.Here is an example of the application in use:

# ./pdrop_ccm /home/jerry/kernext/pdrop_kernex64 fred 1.0 Extension is loaded, with kmid 1353052160Official name is: fred IP addresses: 9.20.XXX.YYYTarget IP address: 9.20.XXX.YYY.

Total In In dropped PC in dropped Total Out Out dropped PC out dropped0 0 0.00 0 0 0.000 0 0.00 0 0 0.0013138 138 1.05 13283 144 1.0829158 275 0.94 29460 302 1.0343738 423 0.97 44206 467 1.0658320 564 0.97 58952 632 1.0772653 721 0.99 73437 784 1.0787227 872 1.00 88153 926 1.05#



Here, we can see that the host to which packets are to be dropped is fred and 1.0%of the packets are to be dropped. Status updates on total and dropped packets arethen displayed at 10 second intervals.

The test application calls the pdrop_reset_counters() method to reset the kernelextension counters. After 10 seconds, you can see that network activity to the targetsystem is started and the packet statistics are displayed.

Note that the example controls a single target system only. If you run the pdrop_ccmcommand again and specify a different target host name, packets to and from theoriginal host will no longer be dropped.

Java control and monitorA sample Java application, PDrop_jcm.java, is also provided, which demonstrateshow to use the packet drop simulator from the Java environment.

PDrop_jcm.java uses JNI to access the C environment for controlling and monitoringthe kernel extension. The shared library libpdrop_jni.so provides native methodswhich can be called from the Java environment. This shared library is built as part ofthe build process described earlier from the source, pdrop_jni.c.

The JNI wrapper provides the following native methods at the Java level which mapto the functions in the shared library:

public native boolean isExtensionLoaded(String extName); public native boolean setDropAddress(String hostname); private native String getDropAddress(); public native boolean amDropping(); public native int setDropping(boolean doDrop); public native void resetCounters(); public native void setDropHostName(); public native void setDropPC(float d); public native float getDropPC(); public native long getTotalIn(); public native long getTotalOut(); public native long getInDropped(); public native long getOutDropped();

When these native methods are invoked, the corresponding function in the sharedlibrary will be called and the result will be returned to the Java environment.

To build and use this test Java application, first, make sure that you have set thePATH to the Java SDK environment correctly, for example:

export PATH=/usr/java6/bin:${PATH}

Next, set the LIBPATH environment variable to reference the directory where thekernel extension is located, so the shared library libpdrop_jni.so can be resolved:

export LIBPATH=/home/jerry/kernext:/usr/lib



Now compile the PDrop_jcm.java application, for example:

javac -d . PDrop_jcm.java

Optionally, if you need to regenerate the JNI header file, run the following command.It mightnot be necessary to run unless you have customized the kernel extension andchanged or added to the native methods:

javah -d . PDrop_jcm

If you have regenerated the header, you will need to rebuild the shared library usingthe process described earlier.

The Java application is then run with the path to the kernel extension, target hostname, and optional new drop rate as arguments, as in the C example above:

# java PDrop_jcm /home/jerry/kernext/pdrop_kernex64 fred 1.0Ext name: /home/jerry/kernext/pdrop_kernex64. Extension is loaded, with kmid 1353052160Drop address is: 9.20.XXX.YYYDrop PC: 1.0Dropping enabledTotal In In dropped PC in dropped Total Out Out dropped PC out dropped0 0 0.0 0 0 0.00 0 0.0 0 0 0.013189 145 1.09 13307 118 0.8828515 295 1.03 28782 267 0.9244306 461 1.04 44701 395 0.8859216 609 1.02 59753 537 0.8972979 756 1.03 73688 709 0.9688606 910 1.02 89455 849 0.94102550 1054 1.02 103567 1017 0.98104522 1071 1.02 105551 1029 0.97104522 1071 1.02 105551 1029 0.97

In this example, you can see that when monitoring started there were zero packetssent or received to or from the target host, but there was then network activity thatcaused packets to be dropped in both directions. In this example, the drop rate hadbeen configured at 1%.

Measuring non-simulated packet loss

It was mentioned earlier that operating system utilities should not be used to monitorpackets dropped through the use of the kernel extension. However, these will berequired when you need to access packets that are actually dropped across thenetwork. This section gives some useful hints and tips for using these utilities.

The ping utility displays the packet loss on the ICMP packets being sent across thenetwork. You can see the packet loss statistics in the above example, where it iszero. However, this measurement is only based on these packets being transferredfrom and to the ping utility and do not measure any other packets traversing thenetwork.



On some systems, ping also supports an option to write ICMP packets as fast aspossible onto the network and again gives you the loss statistics at the end. This is aso called flood ping. You should use this with caution as it is likely to impact generalnetwork performance while running. It should be run only for a few seconds and onlyunder test conditions.

The netstat utility can also be used to see how many packets have been dropped ona network. Here is an example output from netstat -D on AIX.

Using netstat to report packet drops (AIX):

# netstat -D

Source Ipkts Opkts Idrops Odrops-------------------------------------------------------------------------------ent_dev0 26820670 16079610 0 0 ---------------------------------------------------------------Devices Total 26820670 16079610 0 0-------------------------------------------------------------------------------ent_dd0 26820670 16079610 0 0 ---------------------------------------------------------------Drivers Total 26820670 16079610 0 0-------------------------------------------------------------------------------ent_dmx0 26820664 N/A 6 N/A ---------------------------------------------------------------Demuxer Total 26820664 N/A 6 N/A-------------------------------------------------------------------------------IP 50335154 44013816 1110616 147099IPv6 1473 1473 0 0TCP 43110957 40986657 6235 0UDP 6105902 2045877 5022268 0 ---------------------------------------------------------------Protocols Total 99552013 87046350 6139119 147099-------------------------------------------------------------------------------en_if0 26820664 16079528 0 0lo_if0 27970975 27975366 4652 0 ---------------------------------------------------------------Net IF Total 54791639 44054894 4652 0-------------------------------------------------------------------------------NFS/RPC Client 23 N/A 0 N/ANFS/RPC Server 0 N/A 0 N/ANFS Client 14949 N/A 8 N/ANFS Server 0 N/A 0 N/A ---------------------------------------------------------------NFS/RPC Total N/A 14977 8 0-------------------------------------------------------------------------------(Note: N/A -> Not Applicable)#

On AIX, the netstat -Zs -p tcp command can be used to reset the protocolstatistics before running the promotion activity.

If packet drops are consistently in excess of 0.1%, then you should raise this withyour network administrator.

Retransmitted packets can be seen using the following command:

$ netstat -s -p tcp | grep retrans



Statistics of interest from the output of this command are:

Packets sent Data packets Data packets retransmitted Packets received Completely duplicate packets Retransmit timeouts

Conclusion

This article provided reference material and instructions to build, use, and customizea simple utility to simulate dropped TCP packets on AIX. Such a utility is invaluablewhen writing cross network software to model how it will behave under non-idealnetwork conditions.

The tool may be adapted as required and can easily be enhanced to supportsimulation of other network issues that can give rise to performance problems, suchas packet corruption, packets arriving out of order, and jitter.

Acknowledgements

The author would like to thank the following people for helping to review this article:

Kavitha Baratakke, IBM Systems and Technology Group, Systems SolutionDevelopment, IBM USA.

Joshua Carr, WebSphere Service Registry and Performance, IBM UK. Sukesh Chulliyote, Sr Quality Assurance Engineer, Oracle Corporation Nigel Griffiths, IBM Sales and Distribution, Systems and Technology Group

Sales, IBM UK. Paul Harris, WebSphere MQ Performance, IBM UK.



Downloads

Description Name Size Downloadmethod

Packet drop source aix_pdrop.tar.gz 8 KB HTTP

Information about download methods



Resources ipfilter a packet filtering tool written by Darren Reed. IBM developerWorks article on developing AIX kernel extension Follow JNI developerWorks article



About the author

Jerry Stevens

Jerry Stevens leads the WebSphere Service Registry and Repository(WSRR) Performance team at the IBM Software Lab in Hursley Park,United Kingdom. He joined IBM UK Global Services in 1997 as an IBMAIX Consultant, then transferred to WebSphere MQ Development andPerformance Analysis in 2001, and to WSRR Development in 2009.He took on the WSRR Performance team role in 2011. Prior to joiningIBM, Jerry worked for Shell UK as a Senior Systems Engineer, wherehe undertook a range of technical consultancy and development rolesand worked with a variety of open systems platforms and architectures.He graduated from Exeter University with an Honours Degree inMathematics.

Copyright IBM Corporation 2013(www.ibm.com/legal/copytrade.shtml)Trademarks(www.ibm.com/developerworks/ibm/trademarks/)

Table of ContentsIntroductionBackgroundHow to build and use the kernel extension as suppliedRequirementsPrecautionsBuilding the kernel extension and utilitiesLoading and activating the kernel extensionInitiating packet dropping and monitoring

How the kernel extension worksOutbound traffic flowInbound traffic flowInbound filterOutbound filterRandomizing the packets to be droppedSystem calls provided in the packet drop kernel extension

Configuring the kernel extensionAIX 6.1 and AIX 7.1 build issuesThe build processExporting the provided system callsUse of kctrl to control loading of the kernel extensionLogging in the kernel extensionUse with other IP protocols

Control and monitor applicationsC control and monitorJava control and monitor

Measuring non-simulated packet lossConclusionAcknowledgementsDownloadsResourcesAbout the author

Au Aix Packet Dropping PDF

Documents

Transcript of Au Aix Packet Dropping PDF