Flutterby™! : Justified 2011-08-30 10:19:58.204168+00

Justified

2011-08-30 10:19:58.204168+00 by meuon 6 comments

Murder. It should be justified if not required when: IT Director and minions have been delivering CSV files with variable table and field names for months as only option of exporting 2+gb of data from "proprietary legacy system"... AND when you finally watch them do an export it's via hand typed queries on a MySQL 4.1 server, on Windows, via some oddball GUI.

Temporary sanity caused by "mysqldump" and nice clean data is all that saved them, and me.

It was the late 90s. We were a small web-hosting company that also did website development. One customer (around 50-100 employees) wanted their sales force (who traveled around the country) to generate their own database reports via the website. Their IT department would only give us pre-generated reports which required us to parse the file and repopulate a database on our end.

And we couldn't get anything other than a pre-generated report (I suspect there was a large turf war going on inside that company).

other_todd: I always loved the "SQL View" in Access. It's a great tool for teaching joins in SQL. Build the joins visually, change to SQL View, see the code! Access even did a decent job generating the SQL. The real fun comes when you edit the SQL code and then switch back to the GUI (Design View).

The real problem with Access was that it worked a little too well. Some things, like the Design View query builder and the Report designer, worked better than almost anything else on the market. Where Access really fell off was that it was too easy to build applications that almost, just barely, supported multiple users.

I made a decent amount of cash in the 90s tuning and rewriting Access applications typically developed by "the boss". Just about the time the application became critical for day-to-day business, Access would just give up. I'd routinely walk into a business and hear "Everyone out of Access. We need to rebuild the database." That was music to my ears!

95% of those problems could be fixed by breaking the bindings between forms and tables. Simply add VBA code to preload the form in the OnLoad event and write out any changes on Submit (or whatever).

The one caveat was if I had to do a full VB rewrite but they wanted their Access-style reports. Crystal Reports never could reproduce some of what Access would do.

I think I just got myself in trouble (again) by assuming that I could fulfill a request from the boss's boss's boss to share data by simply giving the URL for the XML file (along with another URL explaining the format of that XML file - it's a very big XML file with very few unique object types so it's better handled as plain text).

The problem: the guy trying to use the data, who is well above my pay grade at the NGA and has been on Federal standards committees and is a PhD student at a decent school, only understands spatial data when it can be easily opened in ArcGIS.

Sadly lacking meuon -- you neglected the torture that should precede the execution. Naturally all IT members should be required to view the proceedings.

I think a lot of people deliberately don't want to know what goes on under the hood, and while I'm not biased against fancy UIs - they have many useful and valuable properties - I think UIs do hide that. SQL is 90% the same SQL anywhere you use it, and yet this common coin - the saving grace for anyone who, like me, has to work in three or four different flavors of DB on a regular basis - is kept tucked way out of sight. On the other hand, I think that for some people (perhaps many people) knowing what goes on under the hood is a DISadvantage. People like us, we understand the job better if we know something about how the engine works; but for a lot of other people, knowing anything at all about how the engine works just adds confusing complexity they can't process, and makes the job harder because they now not only have to do the job, but have to sort out "information they actually need to do it" from "information that just muddies the water."

I had a customer watch me type up a fairly elaborate table join in SQL (in Access, god help us) and comment on how he could never have done that ... but he was also aware enough to realize that what I had typed directly in SQL was about fifty times faster to do than what he would have done, which was try to set up joins using Access' graphical method of doing so. The thing is, there was nothing at all wrong with Access' method (this was way back in Access 1.0 and I have no idea how it looks now). It involved actually constructing arrows linking different columns in tables - very much like making an E/R diagram - and probably the best possible way to represent joins visually. But it does take time to do. I offered to teach him SQL - which, I maintain, is not hard to learn - and he declined. Whether SQL was hard to learn wasn't the relevant point; he believed in my ability to teach him. But learning it would have messed with his mastery of the GUI method of doing the same thing. He didn't think he was capable of holding both in his head. For all I know, he may have been right.

There is a distinct lack of knowing what goes on under the hood these days. On one hand I think it's great that we can abstract away the irrelevant bits, on the other hand... well... crap like this.

I've had this problem in talking with a few IT people, where our languages don't intersect because they have no clue about architecture, but are also only used to using GUI interfaces on systems which don't script well (ie: Windows).

I've also been looking at some shops which run Windows and been trying to figure out what path they could possibly take to migrate to open source capabilities. The problem is that you need to hire people who are programmers, or at least scripters, as well as mouse-jockeys, and that takes a big leap of faith that it's hard to get an organization to take.