Jump to content

F5 GTM-DNS Sync Group

guru

Entry posted by guru

10 views

 Share

This is to help better explain the purpose of a sync group on the F5 GTM's or otherwise known as BIG-IP DNS.

The following figure shows that, after a configuration change is made on the Los Angeles BIG-IP DNS system, the local big3d process initiates an iQuery connection to BIG-IP DNS sync group members in New York and Europe and advertises the updated configuration to the remote gtmd processes.

DNS_syncgrp.png

Synchronization details

When you configure BIG-IP DNS synchronization, the sync group members share and synchronize BIG-IP DNS configuration objects and metrics data. The following table lists the relevant configuration objects and whether the objects are synchronized.

BIG-IP DNS configuration object Synchronized
Wide IP addresses Yes
Data centers Yes
Servers Yes
Virtual servers Yes
Links Yes
GSLB iRules Yes
Topology records / Regions Yes
Distributed applications Yes
GSLB global settings Yes
GSLB monitors Yes
DNSSEC zones / Keys Yes
DNS zone files Not synchronized by default
Named configuration Not synchronized by default
Listener addresses No
DNS express zones No
DNS cache No

Synchronization group requirements

Before you configure synchronization, you should be aware of the requirements for BIG-IP DNS synchronization group members to communicate and synchronize properly which are found on F5 K13734,  but the high level summary of it is this for the BIG-IP DNS synchronization group members to properly synchronize their configuration settings. 

Verify that the following requirements are in place:

BIG-IP DNS synchronization group members must be running the same software version

  • A BIG-IP DNS device should be running the same software version as other members in the synchronization group. BIG-IP DNS devices that are running different software versions will not be able to communicate and properly synchronize BIG-IP DNS configuration and zone files. For information about displaying the software version, refer to K8759: Displaying the BIG-IP software version.

Synchronization parameters must be properly defined for all members

  • Synchronization must be enabled and each device must have the same synchronization group name. You can define the synchronization parameters by navigating to on BIG-IP DNS 11.5.0 and later: DNS > Settings > GSLB > General

NTP must be configured on each device

  • Before you can synchronize BIG-IP DNS systems, you must define the network time protocol (NTP) servers for all synchronization group members. Configuring NTP servers ensures that each BIG-IP DNS synchronization group member is referencing the same time when verifying the configuration data that needs to be synchronized. You can configure NTP by navigating to System > Configuration > Device > NTP.

Port Lockdown must be set properly for the relevant self IP addresses

  • Port lockdown is a security feature that specifies the protocols and services from which a self IP address can accept traffic.
  • F5 recommends using the Allow Custom option for self IP addresses that are used for synchronization and other critical redundant pair intercommunications. You can configure port lockdown by navigating to Network > Self IPs.

Note: Management-IP address are not compatible with iQuery; you should not use them as server IP addresses in the DNS server list.

Configure the service ports shown in the following table for BIG-IP DNS operation on the specific self IP.

Allowed Protocol Service Service Definition
TCP 4353 iQuery
TCP 22 SSH
TCP 53 DNS
UDP 53 DNS
UDP 1026 Network Failover

For further information on Port Lockdown behavior, please refer to K17333 listed in the Supplemental Information section below.

TCP port 4353 must be allowed between BIG-IP GTM systems

  • BIG-IP DNS synchronization group members use TCP port 4353 to communicate. You must verify that port 4353 is allowed between BIG-IP DNS systems.

Compatible big3d versions must be installed on synchronization group members

  • The big3d process runs on BIG-IP systems and collects performance information on behalf of the BIG-IP DNS system. For metrics collection to work properly, synchronization group members must run the same version of the big3d process. For more information about verifying big3d version information, refer to K13703: Overview of big3d version management.

A valid device certificate must be installed on all members

  • The device certificate is used by the F5 system to identify itself to a requesting F5 client system. The default device certificate, /config/httpd/conf/ssl.crt/server.crt, must be installed on each sync group member. You can verify the certificate validity by navigating to System > Device Certificates.

Configuration review via GUI

Enable synchronization on the system to ensure that the BIG-IP DNS system that is already installed on your network can share configuration changes with other BIG-IP DNS systems that you add to the BIG-IP DNS synchronization group.

  1. On the Main tab, click DNS > Settings > GSLB > General . The General configuration screen opens.
  2. Select the Synchronize check box.
  3. In the Group Name field, type the name of the synchronization group to which you want this system to belong.
  4. In the Time Tolerance field, type the maximum number of seconds allowed between the time settings on this system and the other systems in the synchronization group.The lower the value, the more often this system makes a log entry indicating that there is a difference. Tip: If you are using NTP, leave this setting at the default value of 10. In the event that NTP fails, the system uses the time_tolerance variable to maintain synchronization.
  5. Click Update.

DNS_GSLB_Settings_General.png

When a change is made on one BIG-IP DNS system in the BIG-IP DNS synchronization group, that change is automatically synchronized to the other systems in the group.

Creating a data center on the existing BIG-IP DNS

Create a data center on the existing DNS system to represent the location where the new BIG-IP DNS system resides.
  1. On the Main tab, click DNS > GSLB > Data Centers .
    The Data Center List screen opens.
  2. Click Create.
    The New Data Center screen opens.
  3. In the Name field, type a name to identify the data center.
    Important: The data center name is limited to 63 characters.
  4. In the Location field, type the geographic location of the data center.
  5. In the Contact field, type the name of either the administrator or the department that manages the data center.
  6. From the Prober Preference list, select the preferred type of prober(s).
    Option Description
    Inside Data Center By default, select probers inside the data center.
    Outside Data Center Select probers outside the data center.
    Specific Prober Pool Select one of the Probers from the drop-down list. When you want to assign a Prober pool at the data center level.

    Note: Prober pools are not used by the bigip monitor.

  7. From the Prober Fallback list, select the type of prober(s) to use if insufficient numbers of the preferred type are available.
    Option Description
    Any Available By default, select any available prober.
    Inside Data Center Select probers inside the data center.
    Outside Data Center Select probers outside the data center.
    None No fallback probers are selected. Prober fallback is disabled.
    Specific Prober Pool Select one of the Probers from the drop-down list. When you want to assign a Prober pool at the data center level.
  8. From the State list, select Enabled or Disabled.
    The default is Enabled, which specifies that the data center and its resources are available for load balancing.
  9. Click Finished.

Defining a server on the existing BIG-IP DNS

You must ensure that a data center where the new DNS system resides is available in the configuration of the existing BIG-IP® DNS before you start this task.
You define a new server, on the existing BIG-IP DNS system, to represent the new BIG-IP DNS system.
  1. On the Main tab, click DNS > GSLB > Servers .
    The Server List screen opens.
  2. Click Create.
    The New Server screen opens.
  3. In the Name field, type a name for the server.
    Important: Server names are limited to 63 characters.
  4. From the Product list, select BIG-IP System.
  5. From the Data Center list, select the data center where the server resides.
  6. From the Prober Preference list, select the preferred type of prober(s).
    Option Description
    Inherit From Data Center By default, a server inherits the prober preference selection assigned to the data center in which the server resides.
    Inside Data Center A server selects the probers from inside the data center where the server resides.
    Outside Data Center A server selects the probers from outside the data center where the server resides.
    Specific Prober Pool Select one of the Prober pools from the drop-down list. When assigning the Prober pool at the server level.

    Note: Prober pools are not used by the bigip monitor.

  7. From the Prober Fallback list, select the type of prober(s) to be used if insufficient numbers of the preferred type are available.
    Option Description
    Inherit From Data Center By default, a server inherits the prober fallback selection assigned to the data center in which the server resides.
    Any Available For selecting any available prober.
    Inside Data Center A server selects probers from inside the data center where the server resides.
    Outside Data Center A server selects probers from outside the data center where the server resides.
    None No fallback probers are selected. Prober fallback is disabled.
    Specific Prober Pool Select one of the Probers from the drop-down list. When you want to assign a Prober pool at the server level.
  8. From the State list, select Enabled.
  9. In the BIG-IP System Devices area, click Add to add a device (server).
    1. Type a name in the Device Name field.
    2. Type an external (public) non-floating IP address in the Address field.
    3. If you use NAT, type an internal (private) IP address in the Translation field, and then click Add.
    4. Click Add.
    5. Click OK.
  10. From the Configuration list, select Advanced.
    Additional controls display on the screen.
  11. In the Health Monitors area, assign the bigip monitor to the server by moving it from the Available list to the Selected list.
  12. From the Availability Requirements list, select one of the following and enter any required values.
    Option Description
    All Health Monitors By default, specifies that all of the selected health monitors must be successful before the server is considered up (available).
    At Least The minimum number of selected health monitors that must be successful before the server is considered up.
    Require The minimum number of successful probes required from the total number of probers requested.
  13. From the Virtual Server Discovery list, select how you want virtual servers to be added to the system.
    Option Description
    Disabled The system does not use the discovery feature to automatically add virtual servers. This is the default value. Use this option for a standalone BIG-IP DNS system or for a BIG-IP DNS/LTM® combo system when you plan to manually add virtual servers to the system, or if your network uses multiple route domains.
    Enabled The system uses the discovery feature to automatically add and delete virtual servers. Use this option for a BIG-IP DNS/LTM combo system when you want the BIG-IP DNS system to discover LTM virtual servers.
    Enabled (No Delete) The system uses the discovery feature to automatically add virtual servers and does not delete any virtual servers that already exist in the configuration. Use this option for a BIG-IP DNS/LTM combo system when you want the BIG-IP DNS system to discover LTM virtual servers.
  14. In the Virtual Server List area, if you selected Disabled from the Virtual Server Discovery list, specify the virtual servers that are resources on this server.
    1. In the Name field, type the name of the virtual server.
    2. In the Address field, type the IP address of the virtual server.
    3. From the Service Port list, select the port the server uses.
    4. Click Add.
  15. Click Finished.
    Note: The gtmd process on each BIG-IP DNS system will attempt to establish an iQuery® connection over port 4353 with each self IP address defined on each server in the BIG-IP DNS configuration of type BIG-IP. Allow port 4353 in your port lockdown settings for iQuery® to work.
    The Server List screen opens displaying the new server in the list.
The status of the newly defined BIG-IP DNS system is Unknown, because you have not yet run the gtm_add script.

Running the gtm_add script

Before you start this task, you must determine the self IP address of a DNS system in the BIG-IP® DNS synchronization group to which you want to add another BIG-IP DNS.
You run the gtm_add script on the BIG-IP DNS system you are adding to your network to acquire the configuration settings from a BIG-IP DNS system that is already installed on your network. For additional information about running the script, see SOL13312 on AskF5.com (www.askf5.com).
Note: The BIG-IP DNS and other BIG-IP systems must have TCP port 22 open between the systems for the script to work. You must perform this task from the command-line interface.
  1. Log in as root to the BIG-IP DNS system you are adding to your network.
  2. Run this command to access tmsh.
    tmsh
  3. Run this command to run the gtm_add script
    run gtm gtm_add
    1. Press the y key to start the gtm_add script.
    2. Type the IP address of the BIG-IP DNS system in the synchronization group to which you are adding this BIG-IP DNS system.
    3. Press Enter.
    4. If prompted, type the root password.
    5. Press Enter.
The BIG-IP DNS system you are installing on your network acquires the configuration of the BIG-IP DNS system already installed on your network.

Implementation result

The new BIG-IP® DNS system that you added to the network is a part of a BIG-IP DNS synchronization group. Changes you make to any system in the BIG-IP DNS synchronization group are automatically propagated to all other BIG-IP DNS systems in the group.

 

Troubleshooting BIG-IP DNS sync connections (11.x - 16.x)

tmsh 

The tmsh utility lists failing server objects as Offline and a failing iQuery connection as Not Connected. The following table lists tmsh commands that you can use to check the status of BIG-IP DNS synchronization group members and iQuery connections.

tmsh component Description Example commands
server Summary of defined DNS/GTM server objects tmsh list /gtm server all

tmsh show /gtm server all

iquery Summary of iQuery statistics tmsh show /gtm iquery all
gtm Summary of DNS/GTM statistics tmsh show /gtm

Note: All members that participate in the iQuery mesh must be listed in the Server List. If a member of the iQuery mesh is not included in the Server List, it may result in some or all monitors intermittently or consistently failing. The monitors fail any time the big3d agent on the missing member (server) is expected to perform and report the monitor status. This can result in virtual servers being marked offline with a reason of no reply from big3d: timed out.

Verify required configuration elements for synchronization group members

For BIG-IP DNS synchronization group members to communicate and synchronize properly, you must verify that certain requirements are in place. To do so, review the following checklist.

Sync requirement Description Configuration utility location tmsh
Software versions Run the same software version for synchronization group members System > Software Management tmsh show /sys software
Sync settings Use the same synchronization group settings for all members DNS > Settings > GSLB > General (BIG-IP 11.5.0 and later)

System > Configuration > Global Traffic > General (BIG-IP 11.4.1 and earlier)

tmsh list /gtm global-settings general all-properties
NTP Configure NTP for all members System > Configuration > Device > NTP tmsh list /sys ntp servers
Port Lockdown Use the Allow Default option for self IPs that process iQuery traffic Network > Self IPs tmsh list /net self allow-service
iQuery port Verify that TCP port 4353 is allowed on interconnecting devices Not Applicable Not Applicable
big3d versions Run the same big3d version on all members.

Note: The big3d version should not be older than the host BIG-IP version.

Not Applicable big3d -v

/shared/bin/big3d -v

Review log files

Reviewing the log files is one way to determine the cause of synchronization/iQuery connection issues. The system logs global traffic events to the /var/log/gtm file. Some of the logging related to synchronization/iQuery connection issues is as follows:

 

Device certificate messages

The BIG-IP system uses SSL certificates for inter-device communication using the iQuery protocol. If device certificates are missing, expired, or contain duplicate common name (CN) entries with certificates on one of the synchronization group members, the system is marked Offline and logs an error message to the /var/log/gtm file that appears similar to the following example: 

SSL error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed

When creating or renewing BIG-IP device certificates, use the following guidelines:

  • Device certificates should have unique and meaningful subject data. For example, the CN field should match the hostname of the BIG-IP system in which the certificate was created.
  • When possible, create device certificates with an extended expiration date.
  • Make sure that SSL certificates are not expired.

 

iQuery Connectivity messages

The iQuery protocol uses TCP port 4353 to connect to synchronization group members. The system logs a successful iQuery connection to the /var/log/gtm file.

For example: 

gtmd[8472]: 011ae020:5: Connection in progress to <iquery_peer>
gtmd[8472]: 011ae01c:5: Connection complete to <iquery_peer>. Starting SSL handshake
gtmd[11895]: 011a5003:1: SNMP_TRAP: Server /Common/<hostname> (ip=<iquery_peer>) state change red --> green
gtmd[11895]: 011a5008:1: SNMP_TRAP: BIG-IP GTM /Common/<hostname> (<iquery_peer>) joined sync group default

If the iQuery protocol is blocked; for example, by a router ACL, or packet filter, the BIG-IP DNS system marks its iQuery peer as Unavailable and attempts to reestablish the iQuery connection every 10 seconds. When this behavior occurs, a log sequence appears in the /var/log/gtm file that appears similar to the following example: 

gtmd[11895]: 011a500c:1: SNMP_TRAP: Box <iquery_peer> state change green --> red (Box <iquery_peer> on Unavailable)
gtmd[11895]: 011a5004:1: SNMP_TRAP: Server /Common/<hostname> (ip=<iquery_peer>) state change green --> red (No communication)
gtmd[8472]: 011ae020:5: Connection in progress to <iquery_peer>
gtmd[8472]: 011ae020:5: Connection in progress to <iquery_peer>
gtmd[8472]: 011ae020:5: Connection in progress to <iquery_peer>
gtmd[8472]: 011ae020:5: Connection in progress to <iquery_peer>

NTP messages

The Synchronization Time Tolerance setting specifies the number of seconds that one system clock can be out of sync with another system clock in the synchronization group. If the time difference between synchronization group members is greater than the Synchronization Time Tolerance value, the system logs a message to the /var/log/gtm file that appears similar to the following example: 

gtmd[11895]: 011a0022:2: Time difference between GTM /Common/B3900-242 and me is 486 seconds -- Make sure NTP is running and GTM times are in sync

This error message is an indication that NTP may not be configured on one or more synchronization group members.

 

Troubleshoot iQuery connectivity

BIG-IP DNS systems in a synchronization group create an iQuery mesh across synchronization group members. For example, the local BIG-IP DNS system's gtmd process opens an iQuery connection to its own big3d process, and to remote synchronization group member's big3d process. There may be occasions when you must test iQuery connectivity between synchronization group members. For example, if log messages indicate that a BIG-IP DNS system has marked its iQuery peer as Unavailable, you can perform the following troubleshooting procedure to test TCP port 4353 connectivity:

Impact of procedure: Performing the following procedure should not have a negative impact on your system.

Log in to the command line.

To verify the iQuery connection status, enter the following netstat command: netstat -na |grep 4353

The following netstat output indicates that the local system (10.11.16.238) is listening on port 4353 and has an iQuery connection established to its own big3d process. In addition, the local system and its iQuery peer (10.11.16.242) have established an iQuery mesh: 

Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 :::4353 :::* LISTEN
tcp 0 0 ::ffff:10.11.16.238:52794 ::ffff:10.11.16.238:4353 ESTABLISHED
tcp 0 0 ::ffff:10.11.16.238:4353 ::ffff:10.11.16.242:58779 ESTABLISHED
tcp 0 0 ::ffff:10.11.16.238:4353 ::ffff:10.11.16.238:52794 ESTABLISHED
tcp 0 0 ::ffff:10.11.16.238:46882 ::ffff:10.11.16.242:4353 ESTABLISHED

If the synchronization group iQuery mesh is incomplete, you can use the iqdump command to determine if the iQuery packets arrive at the destination.

If the iQuery channel is not established, iqdump returns with an SSL error similar to the following example: 

iqdump 10.10.10.20

46947856243768:error:14090086:SSL routines:SSL3_GET_SERVER_CERTIFICATE:certificate verify failed:s3_clnt.c:1168:

Note: If the iqdump command returns a connection refused message, you should ensure connectivity for the iQuery channel is allowed, such as ensuring port 4353 is allowed by the self IP addresses on each system and devices in between. You may need to restart the big3d process to recover from the connection-refused condition.

If the iQuery channel is established, iqdump returns XML similar to the following example: 

iqdump 10.10.10.20

<!-- Local hostname: lc1.example.com -->
<!-- Connected to big3d at: ::ffff:10.10.10.10:4353 -->
<!-- Subscribing to syncgroup: default -->
<!-- Tue May 6 09:55:43 2014 -->
<xml_connection>
<version>11.5.1</version>
<big3d>big3d Version 11.5.1.0.0.110</big3d>

 

Verify device SSL certificates

Each synchronizing group member must have a valid SSL device certificate installed in the /config/httpd/conf/ssl.crt/ directory for iQuery connections to succeed. If log messages indicate an issue with a device certificate on one of the synchronization group members, you can verify the certificate status by performing the following procedure:

Note: SSL certificates signed by a third-party certificate authority (CA) must include both the client authentication (clientAuth) and server authentication (serverAuth) extended key usage (EKU) extensions, to allow use by both server and client applications. For example, the big3d process operates as a server (serverAuth), while the gtmd process operates as a client (clientAuth). For information, refer to K7717: BIG-IP DNS and Link Controller support for third-party SSL certificates.

Impact of procedure: Performing the following procedure should not have a negative impact on your system.

Log in to the command line.

Check the status of the device certificate by entering the following command: 

openssl x509 -noout -text -in /config/httpd/conf/ssl.crt/server.crt

Verify the certificate validity date and confirm whether the certificate is expired.

If necessary, renew the certificate. To do so, refer to K16951115: Changing the BIG-IP DNS system device certificate using the Configuration utility.

Troubleshoot daemons

Impact of procedure: Performing the following procedure should not have a negative impact on your system.

The tmm, mcpd, big3d, and gtmd processes are all critical to synchronizing BIG-IP DNS configurations. To confirm that the daemons are running as expected, use the tmsh command.

For example, to confirm the status of the tmm, mcpd, big3d, and gtmd processes, enter the following command: 

tmsh show sys service tmm mcpd big3d gtmd

If the mcpd process is consuming more than 90 percent of a CPU, and synchronizing actions, such as saving the configuration, may fail. To check the CPU usage for the mcpd process, enter the following command: 

top -p `pidof mcpd`

To quit, enter q.

 

Troubleshoot synchronization group members using the server type

Starting from BIG-IP 12.x, you can use the Server Type field from the tmsh show /gtm iquery command output to determine if the listed BIG-IP DNS devices are fully setup to be in the same BIG-IP DNS synchronization group.

If the BIG-IP DNS device is fully setup to be in the same BIG-IP DNS synchronization group as the remaining listed BIG-IP DNS devices, the command output would have the value of BIGIP-DNS for Server Type as shown in the following example: 

-----------------------------------------------------------
Gtm::IQuery: 192.168.74.129
-----------------------------------------------------------
Server                                                 b100
Server Type                                       BIGIP-DNS

Note: The previous example output is truncated for brevity.

 

If the BIG-IP DNS device is not properly setup to be in the same BIG-IP DNS synchronization group as the remaining listed BIG-IP DNS devices, the command output would have the value of BIGIP for Server Type as shown in the following example:

Important: For remote BIG-IP LTM devices that are integrated into the network with BIG-IP DNS, their Server Type continues to indicate as BIGIP. 

--------------------------------------------------
Gtm::IQuery: 192.168.74.130
--------------------------------------------------
Server                                        b101
Server Type                                  BIGIP

Note: The previous example output is truncated for brevity.

In the case of the BIG-IP DNS device not properly setup, you may want to re-run the gtm_add utility on the affected BIG-IP DNS device again.

 Share

0 Comments

Recommended Comments

There are no comments to display.

Guest
Add a comment...

×   Pasted as rich text.   Restore formatting

  Only 75 emoji are allowed.

×   Your link has been automatically embedded.   Display as a link instead

×   Your previous content has been restored.   Clear editor

×   You cannot paste images directly. Upload or insert images from URL.

×

Announcements



×
×
  • Create New...

Important Information

Privacy Policy

  I accept