Splunk Request Logging
Assumptions:
Log volume will be huge and will only turn for critical applications that too in production. This can be tested in Dev/Pre-prod prior moving to the production but need to be turned off immediately.This will not cause performance issues because of High-speed logging HSL feature.
This logging feature can also be turned on for troubleshooting purposes if required.
Dependencies:
Enterprise splunk team should provision dedicated storage for the new applications with F5 logging feature turned on a permanent basis.
Estimation:
If we turn on logging for all applications in production, current rough estimate would be 1TB logs per day.
Introduction:
Request logging feature is inline F5 feature that replace the functionality of the splunk logging irule. This request logging feature provide metrics such as client-ip, elapsed time, request type & details, response code, pool member etc.These logs integrated to splunk will provide lot of details about a particular application behind a single vip.
Alerts, dashboard and reports can also be generated based of the metrics available per application basis.
These logs will provide lot of help in troubleshooting and understanding more about the application for sustain teams.
Sample Log:
QuoteMar 4 15:57:38 10.47.194.101 "Splunk Logging"|"txsat1slbdv03.thezah.corp"|"1551736658724"|"1551736658724868"|"10.7.156.86"|"63710"|"/Development/vs.dev.103867.aap-api-cit2.thezah.corp"|"10.47.37.242"|"443"|"/Development/pool.dev.103867.aap-api-cit2.thezah.corp"|"10.47.34.35"|"46235"|"10.47.49.66"|"12000"|"GET"|"/sa-health/f5chk.html"|"HTTP/1.1"|""|"Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36"|"304"|"2"|0|"2056"
Description of log fields:-
Time stamp of the log in splunk |
3/4/2019 15:57 |
Device ip sending log |
10.47.194.101 |
descriptive field for splunk queries |
Splunk Logging |
Device_hostname |
txsat1slbdv03.thezah.corp |
Requesttime_milli-sec |
1551736658724 |
Requesttime_micro-sec |
1551736658724868 |
Client_ip |
10.7.156.86 |
Client_port |
63710 |
vip_name |
/Development/vs.dev.103867.aap-api-cit2.thezah.corp |
vip_ip |
10.47.37.242 |
vip_port |
443 |
pool_name |
/Development/pool.dev.103867.aap-api-cit2.thezah.corp" |
snat_ip |
10.47.34.35 |
snat_ip_port |
46235 |
poolmember_ip |
10.47.49.66 |
poolmember_port |
12000 |
http_method |
GET |
http_uri |
/sa-health/f5chk.html |
http_version |
HTTP/1.1 |
referrer |
" " |
user-agent |
Mozilla/5.0 (Windows NT 6.1; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/72.0.3626.119 Safari/537.36 |
http_statuscode |
304 |
Responsetime_milli-sec |
2 |
response_size |
0 |
Responsetime_micro-sec |
2056 |
Procedure:-
Navigate to the following.
Local Traffic -> Profiles -> Other -> Request Logging -> Create as per below fields.
Template contents from image:
"Splunk Logging" | "$BIGIP_HOSTNAME" | "$TIME_MSECS" | "$TIME_USECS" | "$CLIENT_IP" | "$CLIENT_PORT" | "$VIRTUAL_NAME" | "$VIRTUAL_IP" | "$VIRTUAL_PORT" | "$VIRTUAL_POOL_NAME" | "$SNAT_IP" | "$SNAT_PORT" | "$SERVER_IP" | "$SERVER_PORT" | "$HTTP_METHOD" | "$HTTP_URI" | "$HTTP_VERSION" | "$Referer" | "${User-agent}" | "$HTTP_STATCODE" | "$RESPONSE_MSECS" |$RESPONSE_SIZE| "$RESPONSE_USECS"
|
Once logfile is created, apply to virtual server that need this logging feature turned on as per below command.
#tmsh modify ltm virtual /Development/vs.dev.103867.aap-api-cit2.thezah.corp profiles add { logprofile }
Note:- turning logprofile on would not cause any performance issues because of HSL logging feature used, it will only need additional storage assigned from splunk team to get the logs indexed in splunk.
Verification:-
Check logs in F5 index ( index=infra_network ) to make sure transaction logs coming in.
index=infra_network “Splunk Logging” “aap-api-cit2.thezah.corp” latest=+15m
1 Comment
Recommended Comments