Jump to content

Splunk Request Logging


guru

28 views

Assumptions:

Log volume will be huge and will only turn for critical applications that too in production. This can be tested in Dev/Pre-prod prior moving to the production but need to be turned off immediately.This will not cause performance issues because of High-speed logging HSL feature.

This logging feature can also be turned on for troubleshooting purposes if required.

Dependencies:

Enterprise splunk team should provision dedicated storage for the new applications with F5 logging feature turned on a permanent basis.

Estimation:

If we turn on logging for all applications in production, current rough estimate would be 1TB logs per day.

Introduction:

Request logging feature is inline F5 feature that replace the functionality of the splunk logging irule. This request logging feature provide metrics such as client-ip, elapsed time, request type & details, response code, pool member etc.These logs integrated to splunk will provide lot of details about a particular application behind  a single vip.

Alerts, dashboard and reports can also be generated based of the metrics available per application basis.

These logs will provide lot of help in troubleshooting and understanding more about the application for sustain teams.

Sample Log:

Quote

Mar  4 15:57:38 10.47.194.101 "Splunk Logging"|"txsat1slbdv03.thezah.corp"|"1551736658724"|"1551736658724868"|"10.7.156.86"|"63710"|"/Development/vs.dev.103867.aap-api-cit2.thezah.corp"|"10.47.37.242"|"443"|"/Development/pool.dev.103867.aap-api-cit2.thezah.corp"|"10.47.34.35"|"46235"|"10.47.49.66"|"12000"|"GET"|"/sa-health/f5chk.html"|"HTTP/1.1"|""|"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36"|"304"|"2"|0|"2056"

Description of log fields:-

Time stamp of the log in splunk

3/4/2019 15:57

Device ip sending log

10.47.194.101

descriptive field for splunk queries

Splunk Logging

Device_hostname

txsat1slbdv03.thezah.corp

Requesttime_milli-sec

1551736658724

Requesttime_micro-sec

1551736658724868

Client_ip

10.7.156.86

Client_port

63710

vip_name

/Development/vs.dev.103867.aap-api-cit2.thezah.corp

vip_ip

10.47.37.242

vip_port

443

pool_name

/Development/pool.dev.103867.aap-api-cit2.thezah.corp"

snat_ip

10.47.34.35

snat_ip_port

46235

poolmember_ip

10.47.49.66

poolmember_port

12000

http_method

GET

http_uri

/sa-health/f5chk.html

http_version

HTTP/1.1

referrer

"  "

user-agent

Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36

http_statuscode

304

Responsetime_milli-sec

2

response_size

0

Responsetime_micro-sec

2056

 

 

Procedure:-

Navigate to the following.

Local Traffic -> Profiles -> Other -> Request Logging -> Create as per below fields.

image2019-3-7_9-41-21.png?version=1&modi

Template contents from image:

"Splunk Logging"|"$BIGIP_HOSTNAME"|"$TIME_MSECS"|"$TIME_USECS"|"$CLIENT_IP"|"$CLIENT_PORT"|"$VIRTUAL_NAME"|"$VIRTUAL_IP"|"$VIRTUAL_PORT"|"$VIRTUAL_POOL_NAME"|"$SNAT_IP"|"$SNAT_PORT"|"$SERVER_IP"|"$SERVER_PORT"|"$HTTP_METHOD"|"$HTTP_URI"|"$HTTP_VERSION"|"$Referer"|"${User-agent}"|"$HTTP_STATCODE"|"$RESPONSE_MSECS"|$RESPONSE_SIZE|"$RESPONSE_USECS"

Once logfile is created, apply to virtual server that need this logging feature turned on as per below command.

#tmsh modify ltm virtual /Development/vs.dev.103867.aap-api-cit2.thezah.corp profiles add { logprofile }

Note:- turning logprofile on would not cause any performance issues because of HSL logging feature used, it will only need additional storage assigned from splunk team to get the logs indexed in splunk.

 

Verification:-

Check logs in F5 index ( index=infra_network )  to make sure transaction logs coming in.

index=infra_network “Splunk Logging” “aap-api-cit2.thezah.corp” latest=+15m

 

1 Comment


Recommended Comments

  • Administrators

SCENARIO

Walking through troubleshooting since the virtual server will show up, the pool will show up but going to the URL the application doesn't come up as long as on the logging profile you have Respond on error enabled and your logging pool has no available members.

BELOW IS AN EXAMPLE

VIRTUAL SERVER

ltm virtual /Integration/vs.sim1.102799.qa.enterpriseremarketing.int.thezah.com.443 {
    destination /Integration/10.46.65.206:443
    ip-protocol tcp
    last-modified-time 2021-11-15:10:18:54
    mask 255.255.255.255
    partition Integration
    persist {
        cookie {
            default yes
        }
    }
    pool /Integration/pool.sim1.102799.qa.enterpriseremarketing.int.thezah.com.443
    profiles {
        http { }
        logprofile { }
        oneconnect { }
        serverssl {
            context serverside
        }
        ssl.client.qa.enterpriseremarketing.int.thezah.com {
            context clientside
        }
        tcp-lan-optimized { }
    }
    rules {
        /Integration/irule.qa.enterpriseremarketing.int.thezah.com.content.redirect
    }
    serverssl-use-sni disabled
    source 0.0.0.0/0
    source-address-translation {
        pool MOD_SNAT_POOL
        type snat
    }
    translate-address enabled
    translate-port enabled
    vs-index 1492
}

Run a tcpdump against the IP address of the Virtual Server to see what's going on since Virtual Server shows up, Pool shows up

tcpdump -s0 -nni 0.0:nnnp -vvv -w /var/tmp/qa.enterprise_20200312.pcap host 10.46.65.206

In order to utilize the Wireshark F5 plugin, you need to flag the tcpdump command appropriately with -s0 and setting the level of noise by flagging the interface with a colon followed by a single, double, or triple n for, respectively, low, medium, and high details.

request_logging_capture.png

In Wireshark use the display filter: f5ethtrailer.rstcausetxt in order to get the same screen as show above

So from this you can see the F5 is RESETTING the connection (F5RST) and come to find out its because of the below setting in an attached logprofile

request_logging_setting.png

Now the when a virtual server has the logprofile attached it sends an extreme amount of data to Splunk so it only gets turned on critical applications or if you are troubleshooting an application so you can get more log data to help find out whats going on exactly.

With Respond on Error = Enabled it means when the Splunk Servers go down then the application also goes down.  With Respond on Error = Disabled (which is the default setting) then when the logging server goes down, the applications will continue to function.

Easy CLI command to change this to disable is

tmsh modify ltm profile request-log logprofile proxy-respond-on-logging-error no

Hope you find this helpful.  Also note that the wireshark capture above to get the F5RST to be displayed you have to either install the wireshark plugin on Wireshark 2.5 and older and on Wireshark 2.6 and newer you just need to enable it via Analyze - Enabled Protocols - F5 Ethernet trailer - f5ethtrailer

request_logging_wireshark_f5plugin.png

Now you can 

Link to comment
Guest
Add a comment...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

Announcements



×
×
  • Create New...

Important Information

Privacy Policy