Main menu:

Site search

Categories

January 2012
M T W T F S S
« Dec    
 1
2345678
9101112131415
16171819202122
23242526272829
3031  

Tags

Blogroll

Active Directory LDAP integration with RedHat / CentOS / Scientific Linux

This may be a surprise to some people, but I personally believe that Microsoft’s Active Directory is one of the best directory services out there.
It offers LDAP, RADIUS, Kerberos, internal Certificate Authority servers, and more.
Tying all of that together yourself with OpenLDAP, Samba, open source RADIUS servers, and other services would be a huge undertaking, but it can be done.

I won’t get into everything I hate about other Microsoft products, the purpose for this entry is to assist others in finding their way into integrating their Linux logins, NFS home directories, authentication, password policies, etc into an existing Active Directory infrastructure.

The focus of this particular entry will be around a set of components that RedHat released as part of their new 6.x branch, called SSSD (System Security Services Daemon). SSSD aims to simplify the process of integrating different authentication backends into PAM. This way, you won’t necesarily need a PAM module for LDAP, a PAM module for KRB, module for NIS, etc.
PAM will simply call SSSD, which then handles doing lookups against whatever backend of your choice, in this case we will use LDAP to integrate into Active Directory.

For the initial setup, we need to extend the schema for Active Directory to support additional user attributes, like the Unix UserID, GroupID, Shell, Home Directory, and Primary Group.
Go to one of your Domain Controllers, open Server Manager, under Roles, click “Add Role Services”, then select “Identity Management for UNIX”. You will need to install the NIS Server and Administration Tools, you can skip the Password Sync component. Unfortunately, to get the “Unix Attributes” tab in AD, you need to install the NIS Server, it will not give you the Unix attributes tab without it. You will need to reboot the server for the features to be installed, so take that into consideration.

 

 

With the feature installed, you can start populating user and group properties with new Unix attributes. I suggest coming up with a UID and GID numbering convention, you will still have local system accounts and groups on each server, plus accounts in your directory service. This will come bite you in the ass if you don’t have a plan, like when you create a local account, it will select the next available UID, which will include any on the computer, and in the directory. You will need a plan, and to edit your “login.default” to set a high maximum for local UIDs and GIDs.

 

 

 

You now need to install SSSD on your RedHat/CentOS/Scientific Linux server. You will need to have the ‘sssd’ and ‘sssd-client’ packages installed.
Once installed, edit the ‘/etc/sssd/sssd.conf’ file, use the below as an example of entries you need to authenticate to a domain. I also suggest exporting the Certificate Authority from your Domain Controller to ensure trusted/encrypted traffic for LDAP.


[sssd]
config_file_version = 2

# Number of times services should attempt to reconnect in the
# event of a crash or restart before they give up
reconnection_retries = 3

# If a back end is particularly slow you can raise this timeout here
sbus_timeout = 30
services = nss, pam

# SSSD will not start if you do not configure any domains.
# Add new domain configurations as [domain/] sections, and
# then add the list of domains (in the order you want them to be
# queried) to the "domains" attribute below and uncomment it.
domains = domain

[nss]
# The following prevents SSSD from searching for the root user/group in
# all domains (you can add here a comma-separated list of system accounts that
# are always going to be /etc/passwd users, or that you want to filter out).
filter_groups = root
filter_users = root
reconnection_retries = 3

# The entry_cache_timeout indicates the number of seconds to retain an
# entry in cache before it is considered stale and must block to refresh.
# The entry_cache_nowait_timeout indicates the number of seconds to
# wait before updating the cache out-of-band. (NSS requests will still
# be returned from cache until the full entry_cache_timeout). Setting this
# value to 0 turns this feature off (default).
; entry_cache_timeout = 600
; entry_cache_nowait_timeout = 300

[pam]
reconnection_retries = 3

[domain/yourdomain]
debug_level = 0
description = LDAP domain with AD server
enumerate = true

id_provider = ldap
auth_provider = ldap
chpass_provider = ldap

ldap_uri = ldap://dc01.domain.com
ldap_tls_cacertdir = /etc/openldap/cacerts
ldap_tls_cacert = /etc/openldap/cacerts/domain.crt
ldap_schema = rfc2307bis
ldap_user_search_base = DC=domain,DC=com
ldap_group_search_base = OU=Security Groups,DC=domain,DC=com
ldap_default_bind_dn = CN=LDAPBIND,DC=domain,DC=com
ldap_default_authtok_type = password
ldap_default_authtok = somepassword

ldap_user_object_class = user
ldap_user_name = uid
ldap_user_uid_number = uidNumber
ldap_user_gid_number = gidNumber
ldap_user_home_directory = unixHomeDirectory
ldap_user_shell = loginShell
ldap_user_principal = userPrincipalName

ldap_group_object_class = group
ldap_group_name = name
ldap_group_member = member
ldap_group_gid_number = gidNumber
ldap_force_upper_case_realm = True

Gmail and Google Apps TLS broken

While troubleshooting various client reported issues with e-mail delivery to Google apps the last couple days, I have noticed intermittent and undesired behavior from Google’s inbound MX servers.
This lead me to begin testing all the inbound MX records for Google domains, and domains hosted by Google Apps to see if I could reproduce this reliably. I began to think, Google Apps claims it is secure enough for all of your personal and company (event government and education) e-mails.
However, over the last two days, it appears Google has been having widespread, yet intermittent issues with its inbound mail servers not advertising TLS capabilities correctly. While a lot of systems simply use oppurtunistic TLS, meaning they will try TLS if it is available, and if not will continue with plain text delivery, a lot of firms including banks will establish mutual encryption policies between each other requiring TLS encrypted deliveries.
I deal a lot with firms who are either hosted with Google Apps, or conduct a large amount of their business with other firms who have moved to Google Apps, while it is a fantastic platform, contacting support for Google Apps can be an exhausting effort. This issue especially looks bad for the “cloud” in general, because security is such an important part of these shared resources.

To verify this is indeed an issue on Google’s side, and not simply an ISP in the middle filtering port 25 or other protocol anomolies, I performed my test from 4 seperate networks, including my home and business connections with the exact same results.

For instance, we would like to deliver mail to a user at Gmail.com, which the MTA will perform an MX record lookup. Today the MX records are as follows:


gmail.com. 1960 IN MX 10 alt1.gmail-smtp-in.l.google.com.
gmail.com. 1960 IN MX 20 alt2.gmail-smtp-in.l.google.com.
gmail.com. 1960 IN MX 30 alt3.gmail-smtp-in.l.google.com.
gmail.com. 1960 IN MX 40 alt4.gmail-smtp-in.l.google.com.
gmail.com. 1960 IN MX 5 gmail-smtp-in.l.google.com.

Lets emulate a normal SMTP connection to the highest priority (5, in this case):

brent@brent-Ubuntu:~$ telnet gmail-smtp-in.l.google.com 25
Trying 74.125.53.27...
Connected to gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP 3si5619269ano.164
ehlo domain.com
250-mx.google.com at your service, [xxx.xxx.xxx.xxx]
250-SIZE 35651584
250-8BITMIME
250 ENHANCEDSTATUSCODES

In this transcript, STARTTLS is not listed in the capabilities, neither is PIPELINING. Lets try and perform STARTTLS regardless:


STARTTLS
502 5.5.1 Unrecognized command. 3si5619269ano.164

No such luck with TLS on this MX. If this was a real message, it wouldn’t have even tried a TLS delivery unless your mail server was configured to enforce TLS, at which case your mail server may defer the message to retry later, or send a bounce to the original sender saying TLS could not be negotiated and the message will not be delivered.

Some mail servers may try the next lower priority MX record, in this case the priority 10 record. Heres a quick connection to that one:


brent@brent-Ubuntu:~$ telnet alt1.gmail-smtp-in.l.google.com 25
Trying 74.125.95.27...
Connected to alt1.gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP 37si6220649ibi.37
ehlo domain.com
250-mx.google.com at your service, [xxx.xxx.xxx.xxx]
250-SIZE 35651584
250-8BITMIME
250 ENHANCEDSTATUSCODES
250 PIPELINING
STARTTLS
502 5.5.1 Unrecognized command. 37si6220649ibi.37

Again, no such luck. Continuing on to the next one:


brent@brent-Ubuntu:~$ telnet alt2.gmail-smtp-in.l.google.com 25
Trying 74.125.65.27...
Connected to alt2.gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP a51si5669035yhc.75
ehlo domain.com
250-mx.google.com at your service, [xxx.xxx.xxx.xxx]
250-SIZE 35651584
250-8BITMIME
250 ENHANCEDSTATUSCODES
STARTTLS
502 5.5.1 Unrecognized command. a51si5669035yhc.75

Sigh. Next.


brent@brent-Ubuntu:~$ telnet alt3.gmail-smtp-in.l.google.com 25
Trying 74.125.93.27...
Connected to alt3.gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP l33si5425415qck.165
ehlo domain.com
250-mx.google.com at your service, [xxx.xxx.xxx.xxx]
250-SIZE 35651584
250-8BITMIME
250 ENHANCEDSTATUSCODES
STARTTLS
502 5.5.1 Unrecognized command. l33si5425415qck.165

Really Google? Really… We only have one left…


brent@brent-Ubuntu:~$ telnet alt4.gmail-smtp-in.l.google.com 25
Trying 74.125.77.27...
Connected to alt4.gmail-smtp-in.l.google.com.
Escape character is '^]'.
220 mx.google.com ESMTP w5si6118027eeh.12
ehlo domain.com
250-mx.google.com at your service, [xxx.xxx.xxx.xxx]
250-SIZE 35651584
250-8BITMIME
250 ENHANCEDSTATUSCODES
250 PIPELINING
STARTTLS
502 5.5.1 Unrecognized command. w5si6118027eeh.12

We’ve tried all the MX’es that Google lists for Gmail.com. Notice that some servers advertise PIPELINING and some don’t. Who knows why, but it appears like some bad mis-configurations on Google’s side.

I will save some screen space, but I have repeatedly connect to the priority 5 MX enough times, that I finally got STARTTLS to show up in the capabilities (after about 100 attempts).

I then moved on to test my personal domain hosted on Google Apps (brentrjones.com, obviously).
Attempt 1 failed to my Google Apps personal domain :

brentrjones.com. 41903 IN MX 10 ASPMX.L.GOOGLE.com.
brentrjones.com. 41903 IN MX 20 ALT1.ASPMX.L.GOOGLE.com.


brent@brent-Ubuntu:~$ telnet ASPMX.L.GOOGLE.com 25
Trying 72.14.213.27...
Connected to ASPMX.L.GOOGLE.com.
Escape character is '^]'.
220 mx.google.com ESMTP z23si5468306wfd.138
ehlo domain.com
250-mx.google.com at your service, [xxx.xxx.xxx.xxx]
250-SIZE 35651584
250-8BITMIME
250 ENHANCEDSTATUSCODES
STARTTLS
502 5.5.1 Unrecognized command. z23si5468306wfd.138

However, on a second attempt mere seconds later:


brent@brent-Ubuntu:~$ telnet ASPMX.L.GOOGLE.com 25
Trying 74.125.127.27...
Connected to ASPMX.L.GOOGLE.com.
Escape character is '^]'.
220 mx.google.com ESMTP p4si147285ybh.41
ehlo domain.com
250-mx.google.com at your service, [xxx.xxx.xxx.xxx]
250-SIZE 35651584
250-8BITMIME
250-STARTTLS
250 ENHANCEDSTATUSCODES
STARTTLS
220 2.0.0 Ready to start TLS

Success! At this point, I’ve made more than a hundred telnet sessions to Google (to Gmail.com inbound MX’es, and to my personal domain’s Google Apps inbound MX’es). Sadly, PIPELINING was not advertised as a supported capability along with TLS. Though it isn’t a huge issue, it begs the question: Why do Google’s inbound MX servers have such wildly different behavior from one another?

The clearly intermittent behavior unsettles me a great deal. I would hope they would have their mail server configurations identical across the board, and at the very least supporting even the most basic of security, that is however not the case.

The bottom line, if you have a Gmail account or Google Apps hosted domain, your mail has likely been delivered over the last couple days without any opurtunistic TLS, and if someone required TLS to deliver to your domain, chances are you never got it, and likely never will unless they resend it.
I have made attempts to reach out to Google the last couple days to their mail admins without any success, their Tier 1/2 support have been unable to even understand TLS, let alone why someone would maybe want it supported.
This is nothing against Google, I have seen many mis-configured mail servers in my time, but now that Google is entering the enterprise, government and education spaces, security and privacy are even more important.
I won’t pretend SMTP is the most secure protocol either, it is likely one of the least secure ones, but there are many simple steps you can take to provide some layers of security, including TLS, certificate validation, SPF records, etc. But for a vendor the size of Google, uncorrected flaws that last for days that compromise even basic security should absolutely not be tolerated.

I encourage others to perform their own steps similar to those above, if anything for peace of mind for any mail system to ensure it is configured to reasonable levels.

OpenBSD CARP and VRRP conflicts

Turns out, the OpenBSD CARP (failover protocol) conflicts in port number, and behavior with devices running VRRP.
We have switch and router equipment which uses VRRP to perform gateway fail over. When we brought in CARP, using the same CARP VHID, and VRRp VRID will cause severe packet loss, as it seems the devices become confused with multicasts broadcasting conflicting member status information.

Avoid using the same CARP VHID, with the same VRRP VRID on the same ethernet segment :)

OpenSolaris ZFS replication

I’ve had this goal for quite some time now, every since my employer went with Sun X4540 storage systems to serve as our data storage for backup applications. The goal was to handle data replication at the ZFS file system level, removing the need for application-level awareness of the file system replication.
A couple products from Sun seemingly accomplish this, one being ZFS via the ‘zfs send/recv’ function, and the other being the Sun Availability Suite (AVS).
From a technical standpoint, AVS is a very mature product, and has many features above and beyond ZFS send/recv. However, since AVS is largely file system agnostic, it poses some problems when using it to replicate ZFS file systems. Namely, a ZFS resilver is a block level operation, AVS would see this as data change, and replicate the entire resilver process over the network.
For my application, that is an undesirable situation, as we will be replicating the data off-site via a rather expensive private leased network.

This brings us to the native ZFS send/recv options. There are many resources online about how it works technically, so I would suggest reading up a bit as I wont explain here.

I explored the ZFS Timeslider tools, which have the ability for every local snapshot you take, to execute an additional command (such as zfs send/recv via SSH). That worked for a while, but it was not designed to handle ZFS replication as part of the suite. When my snapshot sizes began to grow, the send/recv operation would take longer than the window before the next local snapshot was taken.
This caused the service to always enter maintenance mode, as conflicting operations would happen.

Then I found a help blog by Sun Engineer Constantin Gonzalez (http://blogs.sun.com/constantin/entry/zfs_replicator_script_new_edition)
Where he described and made available a script that would handle parts of ZFS replication, from the initial snapshot, to sending it to a remote hosts over SSH. However, the same issues haunted me there, the send/recv operation would run past the scheduled window, and subsequent jobs would step on each other and cause issues.

Clearly, these tools accomplished a lot of great things, but some additional logic could be added to ensure jobs can run past their window, without risk of additional jobs trying to take snapshots while others are in progress.

Enter ZFS user properties; you can set arbitrary properties on a per filesystem, or per snapshot level. For example, you can “lock” a file system so that your programs will check to see if a flag exists, and if so, quit gracefully and notify you.
Jobs running past their window will always happen, and a simple check to see if an existing job is running on your data set would avoid conflicting snapshots, failed jobs, etc.
Short of using an enterprise job scheduling program like Control-M, this functionality is simple to add to existing shell scripts!

But theres more, why not use the ZFS user properties to assign additional flags, such as flagging it when all operations complete, or if a snapshot depends on another for incremental sends, or if the local snapshot has been replicated fully.

I took examples from the tools previously created (see above), and added some of those checks and flags. I’ve posted the script on my site, in hopes others will find the additions helpfull, and hopefully improve on some of the incorrect ways I’ve done things.
By no means am I great at writings scripts or programs, so if you see any bugs, or improvements you can make, please make them!

Download: replicate.ksh

Again, suggest or make any improvements, and enjoy!

FreeBSD NFS performance and OpenSolaris

While setting up and testing our Sun X4540 OpenSolaris NFS server, I noticed that our FreeBSD NFS clients were having severe performance issues while writing to the server. After a few days of digging around, I came across some ancient posts (circa 2005) on the FreeBSD-performance mailing lists describing similar problems.
Here is a brief explanation of how an NFS write can happen:
Assuming we have a generic NFS server, and a generic NFS client mounted over NFSv3/TCP, for every file write you issue an FSYNC. This will signal the NFS server to write to disk immidiately what it has received from the client, and once it has written to disk it will acknowledge to the client that the write was successful so the client can send more data.
What I ran into was that for every 32KB (the default write size over TCP), the FreeBSD NFS client was issuing FSYNCs. Normally this isn’t too much of a problem for most NFS servers, but the issue we have here is the guarantee from ZFS that the on-disk state must always be consistent. ZFS will treat an FSYNC literally, and commit to disk every FSYNC. Most other file system will lie at some level, and acknowledge the write before the data actually hits the disk. While that can improve performance, if you had a crash or power failure before actually commiting to disk, you may lose data.
For my case, lying would be a good thing, as those FSYNCs cause the performance to suffer dramatically.

We have in the FreeBSD source code the following section on/near line 143:


#ifdef VM_AND_BUFFER_CACHE_SYNCHRONIZED
if (S_ISREG(fs->st_mode) && fs->st_size > 0 &&
fs->st_size <= 8 * 1048576) {
if ((p = mmap(NULL, (size_t)fs->st_size, PROT_READ,
MAP_SHARED, from_fd, (off_t)0)) == MAP_FAILED) {
warn("%s", entp->fts_path);
rval = 1;
}

What this roughly approximates to, that for files smaller that 8MB (8 * 1048576) we use Mmap to handle the file. Larger files will use the native write() function.
Normally, Mmap can increase the speed a little bit for writing many small files at once, hence the behavior to use Mmap for files smaller than 8MB. The drawback appears to be that Mmap will issue FSYNCs for every write size over NFS.

To get around this behavior, you can decrease the file size at which Mmap will attempt to be used, in my case I set it to (8 * 8), which you likely wont come across files that small. Now, pretty much all file copy operations will use write(), which will go much faster over NFS, but you may incur a 10-15% performance hit on local disk copies of small files, to gain almost 100 times speed improvement over NFS. While this still gives you the protection of FSYNCs at file close, it is true that under some circumstances you may lose data on the NFS server if there is a crash mid-write, however ZFS ensures that the on-disk state will always be consistent, so your file system should not have any errors.

The ‘cp’ utility (actually, mv, cp are both the same, just linked) already seems to have this logic built in:
http://src.opensolaris.org/source/xref/onnv/onnv-gate/usr/src/lib/libcmdutils/libcmdutils.h#65

As you can see, it will not use Mmap for really small files (less than 32MB), but it does indeed use Mmap, and will use it in 8MB chunks.

Below are two comparisons, the same hardware, but one compiled with FreeBSD using Mmap for small files, and the other write() for all files.
The data set was 62 Gigabytes of mail, most file sizes are less than 1MB, and commonly about 12KB to 300KB with random dispersion of larger files up to 25MB.

Mmap speed:

# time cp -Rv Maildir/ /mnt/obsmtp/ei_obscan/Maildir01


real 315m5.054s
user 0m8.418s
sys 10m12.513s

Write() speed:

$ time cp -Rv Maildir/ /mnt/obsmtp/ei_obscan/Maildir02


real 199m11.364s
user 0m7.324s
sys 5m39.594s

You can see the obvious performance increase, but it could be greater if file distribution was a bit larger, as for every single file close it will still issue an FSYNC which still slows it down a little bit.
Average data transfer speed went from less than 1MB/second bursts, to over 100MB/sec bursts. Sustained write was limited by FSYNCs on small files.

However, it appears there are other problems with FreeBSD NFS performance, the FreeBSD NFSv4 client always issues an FSYNC for every write block size, regardless if Mmap or write() is used. If you compare the code from the FreeBSD NFSv3 to the NFSv4 clients, you see that NFSv3 has more logic to handle ASYNC or FSYNC writes, and will tend to use ASYNC given the choice. While the NFSv4 client has no such ASYNC code in it from my examination (please correct me if I’m wrong).

I have done some packet snoops of the NFSv4 client, and not a single ASYNC write is ever issued, but mounting the same share via NFSv3 will issue ASYNC writes. Other NFS clients such as the OpenSolaris NFSv4 will use ASYNC and FSCYN appropriately, so this seems limited to only FreeBSD.

I’ll have to bring this discussion to the attention of the FreeBSD people, to see if there any any ways to improve the NFSv4 client, but it seems simply adding ASYNC writes will improve it significantly.

A 552MB ISO being copied with NFSv4 and NFSv3, showing the impact of F

552M JanĀ  2 18:22 7.1-RELEASE-i386-disc1.iso

NFSv4 mounted:
# mount_nfs -4T 10.0.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/


$ time cp 7.1-RELEASE-i386-disc1.iso /mnt/obsmtp/1001/


real 9m39.517s
user 0m0.000s
sys 0m3.349s

And here we have NFSv3 mounted:

# mount_nfs -3T 10.0.0.19:/pdxfilu01/obsmtp /mnt/obsmtp/

$ time cp 7.1-RELEASE-i386-disc1.iso /mnt/obsmtp/1001/
real 0m12.682s
user 0m0.000s
sys 0m1.265s

OpenSolaris 2008.11 NFSv4 client:
# mount -F nfs -o vers=4 10.0.0.19:/pdxfilu01/obsmtp /mnt/obsmtp


$ time cp 7.1-RELEASE-i386-disc1.iso.1 /mnt/obsmtp/101/


real 0m10.997s
user 0m0.008s
sys 0m1.772s

Pretty clear example of how much ASYNC writes improve NFS performance!