RTFM (Read The Fine Manpage) (logrotate)

Here’s another quick one.

I’ve been fighting this one for years, and never had the additional hour to figure out what was really happening.

Why are my logfiles rotating recursively? As in:

dovecot.error.log-20210101-20210201-20210301-20210401-20210501-20210601-20210701-20210801

I just never figured it out. I know I changed stuff around and was eventually able to get it to stop. But I never ran it down all the way to the end.

That’s a problem these days. So many things you don’t understand, and don’t have the time to fix the right way.

The answer turns out to be simple (and is now highlighted in my local copy of the manpage):

Please use wildcards with caution. If you specify *, logrotate will rotate all files, including previously rotated ones.

logrotate(8) (Red Hat Linux)

This means you, trying to shortcut logrotate to process all dovecot.* pages.

Fun with NFSv4 and mount paths

Here’s one that bit me, and chewed off far too many hours.  The only advantage to the process is that I learned some incidental things about FreeIPA, or “RHEL Identity Management”.

Background: CentOS 7.4 originally, and as vanilla as I could possibly keep things.  I’d started setting up IPA earlier this year, and it all was “working” until I decided to make some “improvements”, namely getting home directories automounted.  I started out using /users as the mount point/parent for home directories on the server, but read some advice to use a different directory on the server to actually hold the home directories so that the file server could also automount the users’ homes to the same location – more consistent.

I set this up, moving the directories, and modifying the IPA map to reflect this.  Though it didn’t happen immediately, this turned out to break things, and so for the benefit of anyone else who trips over this:

Seeing messages indicating that the home directory couldn’t be mounted wasn’t an IPA issue:

mount.nfs4: mounting home1.us.kmpeterson.com:/users/kmp failed, reason given by server: No such file or directory

(actually, most apparently: “can’t mount home directory” on login)

And:

Dec 28 16:27:05 client1.us.kmpeterson.com oddjob-mkhomedir[18428]: error creating /home/kmp: No such file or directory

There wasn’t essentially a problem with the map.  By the time I got to this point, I’d reviewed over and over the exports file entry:

/users *(rw,sec=krb5p:krb5i:krb5,fsid=0)

That’s pretty clear.  And mount (on the NFS server) appears to reflect this:

/dev/mapper/volgroup1-home on /users type xfs (rw,relatime,seclabel,attr2,inode64,noquota)
auto.home on /home type autofs (rw,relatime,fd=19,pgrp=1189,timeout=300,minproto=5,maxproto=5,indirect,pipe_ino=31126)
/dev/mapper/volgroup1-home on /home/kmp type xfs (rw,relatime,seclabel,attr2,inode64,noquota)

And the one I really thought was germane:

lookup_read_map getautomntent_r lookup_read_map: lookup(sss): getautomntent_r: No such file or directory

What’s the answer then:

The first message mislead me.  “No such file or directory” indicated the issue was that NFS wasn’t exporting what I thought it was.  The real problem is that the client wasn’t trying to mount what I thought it was.

The export had been changed to export /users rather than /home.  My previously working map, when I was using /home on the server had this option string:

-fstype=nfs4,rw,sec=krb5p:krb5i:krb5,soft,rsize=8192,wsize=8192 home01.us.kmpeterson.com:/home/&

Changing home to user:

-fstype=nfs4,rw,sec=krb5p:krb5i:krb5,soft,rsize=8192,wsize=8192 home01.us.kmpeterson.com:/users/&

Wasn’t the right answer.  The right answer is:

A client using NFSv4 to mount a filesystem exported with fsid=0 won’t process (as understood in my CentOS 7 environment) the “server path”.  This means that the export must be the root, and the mount specifies the path.  The mount key must end with the string :/& — the /home from the server along with the key construct the target path.

Or, what worked is this:

/etc/exports:

/home *(rw,sec=krb5p:krb5i:krb5,fsid=0)

auto.home option string:

stype=nfs4,rw,sec=krb5p:krb5i:krb5,soft,rsize=8192,wsize=8192 home01.us.kmpeterson.com:/&

This mounts home01.us.kmpeterson.com:/home/kmp to /home/kmp on the NFSv4 client.

Clear as… well, it’s been a long day.

Setting up Puppet Again

How things are often: something gets done, and for whatever reason I don’t write up the notes very well.

(Yes, there’s the larger issue of why I actually am duplicating any effort at all, but it’s a long story.  Later.)

I’ve been working on setting up several Puppet installations over several different networks, and I’d actually gotten a simple set of tasks working.  So, I said there’d be no problems in setting up a new network.  After all, my notes said:

Well, now, that shouldn’t be a problem.  Look at 2) .  I’m sure I would have added notes later if that wasn’t true.

Two hours later, the page looked a bit different:

Two additions:

First, my reminder to myself that misleading documentation is worse than none.

Second: needed firewall update, the autosign.conf file wasn’t there, another old config, and the kicker: Puppet wouldn’t start – just like when I when I worked this the first time in March, and that took some time to figure out, again.

It turns out that Puppet (server) needs access to temporary space that’s generally allocated in /opt/puppetlabs (I think).  The problem is that our security baseline has all filesystems except for those with system-supplied binaries (so, places like /data which is where we put configuration and payloads for services, /tmp, etc.), mounted with the noexec option, meaning no file on the filesystem can be executed.  Puppet’s default temporary allocation apparently needs to write and execute something there, which breaks Puppet Server startup.

My recollection of this is that it’s only an issue during initialization, so I decided the cleanest place to allocate the space was in /boot.

(Meta-apology: I should have documented better how exactly I came to this conclusion, but what makes this meta is that since I had to go back unexpectedly and re-hash this, I jumped to the conclusion – not recording what led me to it.  But now that I’ve set up this category for just that type of thing, I’ll help someone worthy  – probably me, in 2019 – with the next thing I run into.)

Ouch!

So, here’s hoping that Google &co. are running down these pages…

I realized a while ago that I’ve run into too many intractable issues – except that I keep thinking, “this can’t just be me”.  And there are legions of people out there with blogs, and Google finds on them answers to …well, sometimes those very specific intractable problems.

Thanks to all you out there.  Now: because I’m still working on relatively meaty problems, it’s time to give back.

I’m setting up the Ouch category to highlight things that bit me.  I’m going to try to put out there a post when I run into something that I couldn’t find an answer to but had to figure out myself.

Here’s a stupid one: only because it seems reasonable now but I had to figure this out twice: setting up a new Puppet server (next).