10 February 2015

Finding Orphaned Data Files

So Kendra Little's post about how dropping a database that is in an offline state will not clean up the MDF and LDF files got me thinking. I hadn't really considered this before, and wanted an easy way of seeing if I was wasting space with large MDF files that we thought we dropped years ago...a query across CMS should fit the bill.

Here's what my code does:
  • Extracts the folder path and data file names for all databases on the instance, from sys.master_files.
  • For each unique path that is used for storing data files, use xp_cmdshell to get all file names stored there.
  • Find any MDF or NDF files in that list that aren't mentioned in sys.master_files.

Here's the code:

DECLARE @files TABLE (pth VARCHAR(500), fname VARCHAR(500));
DECLARE @cmdOutput TABLE (txt VARCHAR(1000) NULL);
DECLARE @cmd VARCHAR(1000), @MDFpath VARCHAR(500);
SET NOCOUNT ON;

INSERT INTO @files
SELECT
        
LEFT(physical_name,LEN(physical_name)-PATINDEX('%\%',REVERSE(physical_name))+1),
        
RIGHT(physical_name,PATINDEX('%\%',REVERSE(physical_name))-1)
FROM sys.master_files WHERE type = 0;

DECLARE pathcurs CURSOR FOR SELECT DISTINCT pth from @files
OPEN pathcurs
FETCH NEXT FROM pathcurs INTO @MDFpath
WHILE @@FETCH_STATUS = 0  
BEGIN
        SET
@cmd = 'dir ' + @MDFpath + ' /a:-d /b';
        
INSERT INTO @cmdOutput EXECUTE master..xp_cmdshell @cmd;
        
FETCH NEXT FROM pathcurs INTO @MDFpath
END  
CLOSE
pathcurs
DEALLOCATE pathcurs

SELECT c.txt AS DetachedDataFiles FROM @cmdOutput c
LEFT JOIN @files f ON UPPER(c.txt) = UPPER(f.fname)
WHERE UPPER(c.txt) LIKE '%.[M,N]DF%' AND f.fname IS NULL;


Note, in SQL 2005 you'll probably see references to distmdl.mdf and mssqlsystemresource.mdf, which are the data files for the Distribution and Resource databases, respectively. You can ignore these, or alter the script above to ignore them.

02 February 2015

PRINT Statement "Churn"

So I was sitting in one of those post-mortem type meetings, the sort fraught with tearful recriminations and teeth-gnashing (or at least, perhaps, they ought to be) and I was amused to observe the birth of a silly superstition. It was a mix of technical and nontechnical folk, business analysts, developers, business VPs, etc. We were discussing the performance of a particular SSIS package that runs a stored proc, appears to be complete in terms of data, but then sits and runs on the server for hours afterward, when it should have completed and closed. One developer coined the term "churn", as in, how butter is made, to describe what SQL Server was doing.

It was baffling to see how readily and happily the non-technical folk (and some of the technicals, for shame) accepted this as a good enough answer. Instead of asking, why is this happening and how do we fix it, the execs just came to view "The Churn" as a necessary part of the data processing. "So, how long do we expect The Churn to take with this large a file?" "Can you get me an ETA for when The Churn will be finished?" Soon everybody was getting in on it...even people on the business side could understand The Churn. I mean, what could be more natural than that, you run an SSIS package, it works, and then you have a few hours for the DTEXEC process to sit and "churn"?

This kind of silly superstition, where we take something we don't understand and invent a name for it as a kind of acceptance, hobbles an IT organization, papering over ignorance instead of exposing it and digging deeper. It turns out in this case, as our appdev manager surmised, it was massively looped PRINT statements (completely unnecessary of course) in the stored procedures.

To demonstrate, I created a sample stored procedure that loops 100,000 times:

CREATE PROC usp_CHURN
AS
BEGIN
   SET NOCOUNT ON
;
  
DECLARE @i INT = 1;
  
DECLARE @text VARCHAR(200);

  
WHILE(@i < 100000)
  
BEGIN
       SELECT
@text='You are at increment number ' + CAST(@i AS VARCHAR(20));
      
--PRINT @text
      
SET @i += 1;
  
END
   INSERT INTO
TempTable SELECT GETDATE()
END


As is, with the PRINT statement commented out, this runs very fast (subsecond), as you would expect. The INSERT just lets you know when the proc is actually done.

So I created a DTSX package that does nothing more than run this process, and from a different server, created a batch file that outputs the time, then runs the package, then outputs the time again. Running the proc as is, well, as I say, there's nothing to it, it's just memory getting moved around. But when I uncomment the PRINT statement, things take a bit of a nosedive.

Comparing the package start time (10:41:33.49) to the timestamp logged in the table (10:41:35.010) we see that less than 2 seconds elapsed in the actual execution of the procedure. However, the package completed at 10:45:23.38. So two seconds to do the work, and almost four minutes for SSIS to handle all those PRINT statements getting pushed back up the chain. Now, imagine that scaled up to millions or billions, and you can understand why we had an apparently complete SSIS package chewing up one of the vCPUs on a box for days. Yes, days. And taking the PRINT statements out fixed it.

Tactical moral of the story: be very sparing and cautious when sticking PRINT statements into heavily looped sections of your code, unless absolutely necessary. Strategic moral of the story: stop accepting surface level explanations and lazy superstitions...dig in and figure it out.

28 January 2015

Finding SID Mismatches from SQL Server to Active Directory

So, I had an issue yesterday where a user couldn't log in to a few specific SQL Server instances; they were getting the following error:


The actual recorded login error in the logs was this:
Error: 18456, Severity: 14, State: 11.
Login failed for user 'Dom\User'. Reason: Token-based server access validation failed with an infrastructure error. Check for previous errors.


Eventually I came upon some rather useful resources for this cryptic error, and took some of the existing scripts and rolled it into a new script that takes all the domain user logins, gets the binary SID stored in SQL, maps it to SDDL format, and pulls the equivalent Windows SID via xp_cmdshell and WMI. Then it compares, and outputs the mismatches. This script isn't particularly well tested but it seemed to go over reasonably well on 2005, 2008R2, 2012, and 2014 instances.

Once you find that you do have a SID mismatch, the unfortunate solution is dropping and recreating the login, whereupon you'll have to make sure all permissions and memberships are retained. Also, one thing I found is that even when there is a mismatch of a SID, it doesn't mean that user can't connect, necessarily. I've yet to fully understand the whole process, but if the user is getting 18456 login failures AND there is a mismatch of SIDs, recreating the login will likely resolve that issue.

Classic disclaimer, don't ever run code you find on the internet on your servers as it will likely truncate all your data, give your servers venereal disease, and contribute to Global Warming. So Caveat Emptor! Also, the SDDL formatting code was stolen borrowed from Todd Engen. The Code:

USE masterGODECLARE @DomName VARCHAR(50);SET @DomName = 'YourDomainGoesHere';IF OBJECT_ID('tempdb..#LoginSIDs') IS NOT NULL DROP TABLE #LoginSIDs;CREATE TABLE #LoginSIDs(
  
name NVARCHAR(128),
  
NTname NVARCHAR(128),
  
BinSID VARBINARY(100),
  
SQLSDDLSID VARCHAR(200) NULL,
  
WinSDDLSID VARCHAR(200) NULL
);
/* Get relevant logins and SQL SIDs */INSERT INTO #LoginSIDs (name, NTname, BinSID)SELECT
  
name,
  
REPLACE(name,@DomName+'\',''),
  
SUSER_SID(name)FROM master.sys.server_principalsWHERE [type] = 'U'  /* currently omitting NT groups */AND name LIKE @DomName+'\%'ORDER BY name ASC;/* Cursor through each to get reformatted SQL SID and Windows level SID */DECLARE @LogName NVARCHAR(128), @BinSID VARBINARY(100);DECLARE sidcursor CURSOR FOR SELECT NTname, BinSID FROM #LoginSIDsOPEN sidcursor   FETCH NEXT FROM sidcursor INTO @LogName, @BinSIDWHILE @@FETCH_STATUS = 0   BEGIN
  
/* Convert binary SID to SDDL format */
  
DECLARE
      
@StringSID VARCHAR(200),
      
@i INT,  @j INT,  @val BINARY(4),
      
@WMIcmd VARCHAR(200),
      
@WinSID VARCHAR(200);
  
SELECT @StringSID = 'S-'
      
+CONVERT(VARCHAR, CONVERT(INT, CONVERT(VARBINARY, SUBSTRING(@BinSID, 1, 1))));
  
SELECT @StringSID = @StringSID+'-'
      
+CONVERT(VARCHAR, CONVERT(INT, CONVERT(VARBINARY, SUBSTRING(@BinSID, 3, 6))));
  
SET @j = 9;
  
SET @i = LEN(@BinSID);
  
WHILE @j < @i
  
BEGIN
       SELECT
@val = SUBSTRING(@BinSID, @j, 4);
      
SELECT @StringSID = @StringSID + '-'
          
+CONVERT(VARCHAR, CONVERT(BIGINT, CONVERT(VARBINARY, REVERSE(CONVERT(VARBINARY, @val)))));
      
SET @j = @j + 4;
  
END;

  
/* Get Windows SID from WMI */
  
CREATE TABLE #cmdOutput (txt VARCHAR(255) NULL);
  
SET @WMIcmd = 'wmic useraccount where (name=''' + @LogName + ''' and domain='''+@DomName+''') get sid';
  
INSERT INTO #cmdOutput EXECUTE master..xp_cmdshell @WMIcmd;
  
SELECT @WinSID=REPLACE(txt,CHAR(13),'') FROM #cmdOutput WHERE txt LIKE 'S-%';
  
DROP TABLE #cmdOutput;

  
/* Update temp table */
  
UPDATE #LoginSIDs
  
SET
      
SQLSDDLSID = @StringSID,
      
WinSDDLSID = @WinSID
  
WHERE NTname = @LogName;

  
FETCH NEXT FROM sidcursor INTO @LogName, @BinSIDEND  
CLOSE
sidcursorDEALLOCATE sidcursor
--SELECT * FROM #LoginSIDs;

/* Output mismatches */
SELECT * FROM #LoginSIDs WHERE SQLSDDLSID <> WinSDDLSID;



test

25 January 2015

Ermahgerd It's Ermaha!

We decided to take a rather spontaneous trip up to Beautiful Omaha, Nebraska. I capitalize Beautiful to denote it is more a term of respect or courtesy than my own personal observation. Still, I hail from Kansas City which is not exactly...anyway, I'll let that line of thought die on the vine and move on to our trip. We came up as a mini-vacation for Debra's birthday. Saturday we went to the Omaha Zoo.


It's a neat zoo, and the geodesic dome for the desert exhibit is impressive.


A wallaby gazing down at us.


Some sort of pig variant, not the most sweet smelling chaps.


Pete is a good sport for the camera...as long as you immediately let him look at the picture on your phone afterward. I guess that's what made the Polaroid corporation all their money.


Spider crabs in the aquarium. Tasty perhaps, but a bit on the creepy side.


Turtle swimming over us in the aquarium tunnel...sharks, rays, tuna (I think), it was quite impressive.


Not the biggest sharks around, but it was nice to see them.


Gretchen watching the rays.


There were several different jellfish tanks, this was some variety of sea nettle.


Another ray happily showing the limitations of your average point and click iPhone photographer.


Gretchen encountering the octopus.


We had a light lunch (the receipt was most decidedly not light) at one of the basic hot dog/hamburger grill joints that seem identical at all zoos. Gretchen wanted to sit on these sea turtles but Random Other Kid seemingly was ready to do battle over it.


Then we headed over to the cats and monkeys area. Some howler monkeys...well, not howler monkeys but something profoundly loud, were bellowing, and Debra assumed they were amplified by a PA...not the case. This one was a beautiful cat.


Jaguar, or as the English call them, Jagg-ee-yarr.


This snow leopard looked irritatingly over at me with a "do you mind?" scowl, then I realized he or she was taking a leak.


Gretchen super excited to see the white tiger.


From another angle, our tiger friend decided to get up and stretch...we thought we were in for a show, and we were, in a sort of way...he laid rather a large present for the zookeepers, and then immediately started licking himself down...there...in classic cat fashion. I refrain from judging, as humans (well...Norwegians, anyway) do have lutefisk.


Malayan Sun bear climbing about.


Ahh, a Salmonella Petting Zoo! Lick your hands clean kids!


I tried challenging this black swan to a fistfight but backed off when he whipped out a flickknife.


Good ol' American black bear. The bear I'm most likely to run into in a social situation in the future, if I hunt out west.


Andean spectacled bear relaxing a bit.


Then we headed back to the hotel. We stopped at the gift shop for a promised inexpensive toy. Pete found his "Star Wars" (not really) binoculars straight away, he's a simple chap and not prone to shopping indecision. He wanted to reenact the Tatooine "Luke searches for R2D2 with binoculars" scene I think. Gretchen on the other hand, wow, getting her to pick something under $10 was quite the event. I eventually started working the sales angle on various stuffed animals and I'm proud to say I closed the sale on a small Bush Baby toy.

One thing I noted at the zoo...the ubiquitous screaming child. Now, I'm not judging, I'm a parent and my kids have never been immune to this kind of spectacular behavioral performance. But compared to the Kansas City zoo, there was definitely an undercurrent of tragedy among the younger set. In the gift shop particularly, the sharp, tired barkings of parents were contrasted against more melting down than in a Zurich fondue restaurant. I guess the largeness of the zoo tends to have kids getting tired out by the end of the day.

Thence to Trader Joe's for some light snacks for dinner, and the hotel. We found out we happened to be right next door to a Cabela's, and I needed socks since we forgot to pack extra, so over to Cabela's, then!


You have to love taxidermy to appreciate this place.


So back to the hotel, where Debra and I enjoyed a bit of Spatlese and now are ushering the kids to bed. Tomorrow, we go to the Strategic Air Command Museum, and from there, to home.

Ahh, but after a night of the sort of sweet repose calculated to dissuade road-tripping parents from future more ambitious plans, we hoisted ourselves groggily up, and began the sport we like to call "keep the children reasonably quiet until it's time to check out". We drove southwest and after 20 minutes or so, arrived at our destination. It was quite cold out, and the family quickly left me in the dust, disappearing here behind the massive Titan, a staggering symbol of the Cold War.


The SR-71 is one of those essential photo ops.


The main hangar hasn't changed much. That's a U-2 overhead.


There was a partially exposed fuselage of a B-25 that we could walk along side, seeing the interior compartments. This was the cockpit.


The B-36 is just enormous, makes a B-52 seem kind of small and svelte in comparison. Six propeller engines, and four turbojets.


Unsurprisingly the kids like the more hands-on exhibits. We stayed over here rather a lot.


But eventually I coaxed them out to see some more aircraft. The B-17, long a favourite and my grandfather's habitat during the brief unpleasantness in Europe during his youth.


Camera got a bit foggy but the kids thought this little parasite fighter (XF-85 Goblin) was rather cute.


But simulators, and anything you can climb about on, were their favorites.


Ahh. Yes. You see that's what's called a reentry vehicle. Err...yes. Bit awkward, this. Well, you see, it was designed to reenter, well, over Russia. In rather not a very pleasant sort of time.


Well, while my daughter begins to ponder the deeper questions of the morality of war, my son seems cool with it and enjoys the interactive Minuteman silo diorama.


Ahh, the B-29. The plane that ended WWII in a particularly controversial way. Not this one, of course, but one of its ilk.


Soft light bathes the ordnance in the 29's forward bomb bay.


An unmanned Apollo vehicle sits in the...hey is that...LUCAS FOR THE LAST TIME GET YOUR CRAP OUT OF MY MUSEUM!!!


Some whirlybirds in the second hangar.


4000lb bomb, amusingly titled "Light" in an ironic twist, perhaps like calling the huge bouncer at the bar "Tiny".


I gotta say, it may not have made as much tactical or strategic sense as the various camoflauge finishes but the stainless or chrome look of this MiG-21 is fetching.


Kids, of course, enjoying the interactive stuff the most.


Then back into the vehicle after an episode of grave indecision in the gift shop, and we headed southeast towards KC, arriving mid afternoon. Now, to fix dinner and prepare for a week of work. Was a fun weekend, and the birthday girl had a good time. Sort of a road-trip sampler, all the fun, tiredness, bickering, and mess of a real road trip but fits in a weekend!

14 December 2014

Chicago SQLskills Trip

So a couple months back we had a short little trip I thought I should blog about. Not a proper, full-measure Neufeldian type affair where when we get back we swear we never want to attempt something so crazy again, but my company was dispatching me for a week of training, and I was fortunate enough to be sent to a SQLskills Immersion event in the outskirts of Chicago. Last year I went to the PASS Summit, which is basically a big SQL Server party of sorts, almost more a social event than a technical one, but this one was straight training...Paul and Kimberly Randal (nee Tripp) expositing in great detail and at great length into the core internals of SQL Server. Exceeded my expectations, I can say, and I enjoyed it rather a lot.

But this blog post is less about that and more about the trip itself, because I finagled the company into letting me drive and hauled my dear family along with, so we made a quasi-vacation of it. The first haul across Missouri was fairly non-descript, at least as I can remember. Probably lots of whining, lots of yelling, lots of requests for windows down, lots of requests for windows up, and such like, but such fades into the tapestry of normal everyday life for us. But we did make a couple stops in St Louis.

I should say, since I am admittedly remiss in keeping the vast, frenzied mob of readers of this blog updated on a regular basis of my hobby du jour, that I am at present enthralled, inspired by my animal-loving and equally carnivorous daughter, to take up, at this late stage, hunting. Gretchen sampled venison tenderloin at a neighbor's house, decided it was the tastiest thing ever, and enlisted me to get her a deer. I was not raised in the hunting tradition, so I'm having to get up to speed rather quickly with it. On that theme, I'm rather taken with elk. Our hemisphere's noblest cervid...I am both in love with these animals and intent someday on hunting them. Hunter's paradox I suppose. Anyway, we stopped by the Lone Elk Park south of St Louis, where they have elk and buffalo. Younger cow here:


The incredible headless buffalo! Kept looking further and further to starboard while we crept by, denying us a proper photograph.


Then off in the distance, we spotted them...the bull!


A bit further along he had quite the little harem maintained it seems.


Another bull browsing by the road, up further.


Thence to the World Bird Sanctuary, right next door. Pleasant little avian zoo with lots of raptors on display, and some more exotic ones. Inside one of the buildings they had a textural table where kids could touch specimens, and Peter is working on his "grip and grin" technique for whitetail bucks.


Out front they have a variety of birds tethered carefully so as to prevent interspecies squabbling, which given these birds' offensive weaponry, would likely be swift and lethal.


First Saturday in September, mark it down! We aren't talking your regional or national type awareness days, this is an INTERNATIONAL awareness day! BE AWARE OF VULTURES ON THAT DAY, WORLD!


Inside the gift shop, amusing little owl chap (not for sale).


The kids love these things. Peter is all grins but Gretchen takes this sort of stage craft seriously and really descends into her characters.


As a bald eagle.


From there over to my teacher's house for a brief social visit. Ustad-ji loves the kids, at least in the measured doses our visits afford. Gretchen found out she was missing a tooth while we were there:


We relaxed in our usual St Louis hotel that night, making an easy night of it, and headed out in the morning for Chicago. We went straight into the city, and hit the Museum of Science and Industry, and interesting hodge-podge of various exhibits strung together somewhat loosely. A very old armoured car from the (previous) turn of the century:


But I admit, most of why I was keen to come to this museum was U-505. Captured U-boat on display inside, with the museum done up to look like a concrete U-boat pen.


Sadly we weren't there in time to do the inside tour (sold out) but maybe later...we could at least walk around the beautiful thing. What a marvel of ingenuity it was, and sobering to reflect on the tonnage that even this ill-fated boat sent to bottom of the Atlantic. I'm not sure but my guess is this is original battle damage on the conning tower...if I recall it was strafed by an aeroplane at some point on its last mission.


Massive screws at the stern.


Two Enigma machines! The loss and cracking of these devices was bad, bad news for the U-boat crews. All of a sudden we started knowing where they were going to show up, had to be disheartening to surface and find a destroyer plowing right down on you with fresh depth charges and hedgehogs at the ready.



Marvellously crafted old periscope...labelling in German, of course.


Deconstructed torpedo. There was almost as much technology packed into one of these weapons as in an entire ship.


Then upstairs and a strange US Navy tribute tucked away, with dioramas of historically important USN ships. Almost as if the museum was slightly embarrassed of its presence...Peter was thoroughly taken with it though. Old Ironsides:


Delivering nuclear winter silently from beneath the waves...SMILE!


And the USS Monitor. Debra was confused by it. It was a strange looking boat, admittedly, but terrifically influential.


Gretchen was super excited to see the chicks (in a farming technology area).


Then out in one of the main halls, you have Spitfires and Stukas dogfighting over various locomotives. A bit more Peter's style as you can imagine.


The model trains were a hit.


As were sitting for long periods on an old streetcar, a surprise star of the proceedings.


Closer look at the Spitfire and the Stuka. I imagine if they got this close together at battle speeds it'd be curtains for them both, but still, nice presentation of two very classic warbirds. I didn't see if they were original or not, but I rather hope so.


From there, over to Indiana to visit Tony Karasek, sitar builder and jawari-wallah to the stars. Very nice chap I knew from the forums, and by reputation among sitar players, agreed to spend the week taking good care of my surbahar and tweaking it, adjusting bridges, etc. His collection has dwindled over the years but still, not a bad showing.


He showed me his new builds, coming right along very nicely.


Then up to Oak Brook to get settled into the hotel. The week was quite lovely for me, spending almost 10 hours a day diving into deep SQL internals subjects and learning at a fantastic pace. They fed us rather nicely, too. Through the week the family visited parks, zoos, and a few local places while I sat in class, although Debra's pictures now reside on a mostly-dead iPhone 3GS, and I'm not sure if I'll be able to extract them. Anyway, we had a grand time, ate out for dinner which is rather a strange luxury for us, and I found the source of my Hungarian smoked bacon just down the street at a Bende store...managed to procure a lovely selection from them. Smoked bacon, a couple type of sausages, ajvar, Tokaji wine (enjoyed after a lot of manual labour and experimentation with methods of extracting a cork without a corkscrew), and cherry palinka, which was a first for me, a sort of Central European kirschwasser. Delightful stuff.


The first night we had a meet-and-greet in the bar and it was fun to have a beer with Paul Randal and Jonathan Kehayias while they discussed my strange SQL Server problem amongst themselves (for future reference, Change Data Capture in 2008R2 recording events in a transaction out of order so that a dependant data warehouse deletes records by mistake...still not resolved and with Microsoft Premier Support). These chaps know their stuff. Here's Paul discussing when he was called in to consult with Myspace when they went from 40 users to 400,000 users almost overnight...design had to change, to scale.


There's something about "smoking oasis", especially when placed in scare quotes, that makes it sound sarcastic and snarky, like it's really just a spot by the loading dock next to the dumpsters, and they know that, but they hate you cause you smoke and they want to rub in the inconvenience by mockingly naming it something that actually sounds pleasant and even idyllic. I don't even smoke and I kind of want to visit The Oasis!


OK, the week drawing to a close, we drove back to pick up the surbahar from Tony. He's a master on these things, and the sound is lovely now, taraf are responsive, and the sustain on the kharaj is unworldly.


The morning we headed back, we defied common sense and headed back into the city to visit the Field Museum. Basically Bass Pro Shops without the retail.


Funny that this dinosaur wasn't even the star of the show to the kids.


The animals were, though. Whitetail in fall.


And whitetail in winter. They look...delicious.


Sort of a small game case. Couldn't drag Gretchen away.


Ahhh, pronghorn antelope. The "speed goat" if you will, second fastest land animal in the world.


Magnificent elk. Like I said, I love these animals. Yes, I also wish to hunt and eat them, but that's as may be, they're still magnificent.


Alaskan browns, and Gretchen doing the modelled stare into space that is her specialty.


CARIBOU GORN. Another favourite of mine.


Miscellaneous Deer case. Was not labelled as such, but with the sika, some "non-typical" species in here, including the tusk-equipped Asian musk deer.


This is the great thing about museums, even when you have a modern exhibit, as long as it is maintained, it will capture that moment in time. For instance, where would you see a computer like this these days? Not even in a pawn shop...only in a museum, at this point. When they put the exhibit together no one was probably considering how quickly the "laptop" itself would become a humorous and rare exhibit.


As an avowed future hunter of elk, this is the spectre that haunts my dreams.


Nice kitty kitty.


Gretchen doing the komodo pose.


It was time to leave if we were going to make Kansas City that night (and we had to rush back to the hotel in hopes that our bag of toys and souvenirs was still next to the parking spot where we hoped we had left it...it was, and great wailing crisis of lost toys was averted) and I wanted to see if I could find the prehistoric Irish Elk, so while the rest of the family browsed the gift shop I skipped through the ancient Americas exhibit, running into this massive Aztec (I believe) disc:


Some interesting masks from the Northwest Indians:


And a towering glade of totems:


Back on the road...we meant to stop in Hannibal this time, but we just kept on straight through. Illinois had some nice rest stops though, with a river snaking lazily past this one:


But we got back in decent time, and it was good to be home. And once back at work, believe me, the DBCC commands did flow.