Month: January 2008

This is a (hopefully more useful) summary/takeaway of my notes from last night.

The folks at NWSource evaluated more than 50 CMSs before narrowing their list down to 2: Drupal & Joomla. Some of their basic factors are familiar to me: cost, use existing programming expertise, “source-available”, customizable in both look and function.  Drupal won, although part of that seems to have been a historical fluke.

This was a very complicated project, which they did (are continuing to do) in pieces. It was the only way they could handle it, but they don’t recommend that approach others. Drupal use will be expanded from nwsource.com to all the other “NW” sites, migrating from several different CMSs/publishing methods.

They used professional services from Lullabot and Fuse IQ, which helped both with technical aspects and with internal acceptance of using open source. There’s a couple of programmers, and a few designers, including one “superstar”, working on the site. The programmers participate as part of the Drupal community, which is different from previous company culture, but they sounded pretty upbeat about the experience.

My biggest disappointment? That the superstar designer who did all the template work wasn’t there. I was really, really, really curious about how they set theirs up.

I think I might take another peek at Joomla for the particular project for which I’ve been considering Drupal, but I also think that Drupal remains a strong contender. (Plus I wouldn’t mind being able to be a helpful volunteer for OlyBlog.) However, unless I can accomplish super-human feats, I probably won’t be able to do a CMS conversion in the time scale I was originally considering. We’ll see.

A few particulars: 

Strengths so far: workflow, custom URLs, taxonomy, photo handling, overall extensibility.

Weaknesses: content input, templates, working with other web software, moving from development to production (particularly with the volume of user-generated content)

Some of the modules mentioned/recommended:  workflow, pathauto, image field, image cache, printer-friendly, RSS (?), services (audience suggestion). They are using upwards of 30 modules, but figure that’s pretty normal.

CCK is apparently incredibly valuable for specialized types of content, and they were very enthusiastic about it.

Questions about performance and security led to a long discussion of tuning measures, including some particular ways to tweak Apache. This was probably the most valuable thing I got out of the evening.

URL rewriting can be very difficult, but they managed to keep all their old URLs even after the Drupal migration!

Best quotes (may not be exact):

• if you can’t get around in the system, any CMS will last you about 2 years.
• everything that we’re relying on is a module
• [follow the community to] mak[e] sure you’re not digging yourself into a hole that no one will be around to get you out of.
•  drupal likes to own everything.
• so happy to have made their [authors & editors] lives better
• if anybody wants to rewrite a module in the drupal way, please do this one. [printer-friendly]
• it’s not drupal, it’s us [their optimization methodology]

they are part of the joint agreement between times & PI. (weird stuff)

“shameless promotion section” – they have an opening for a drupal person. (putting this in for any seattle-area readers.)

first guy (gary?) is product manager. they were thinking about what nwsource IS – used to be a portal! partnered with (?). reformed into a city guide. grew organically – a bunch of things strung together, including in several different languages & database flavors. (pretty much my experiences, too.) it couldn’t be woven together properly with so many different things (articles across 3 different CMSs)

asked everybody (developers) what they’d used, opinions: no particular enthusiasm, so more research. cmswatch report. investigated 50-70; some are easy to disqualify (too expensive, didn’t want pure MS or Java approach) – then experience at various newspapers. “at the van down by the river” experience in boulder, if you needed changes, have to bring the guy a 6-pack. then 2 more (this in Boulder) – last experience was with vignette in a corp-wide initiative, spent millions, 2 years to implement, 2 years running, didn’t have enough money to hire somebody to go any further.

(“if you can’t get around in the system, any CMS will last you about 2 years.” yes!)

not necessarily open-source, but source-available. narrowed down to two: drupal & joomla. joomla “kicked itself out ofthe running early.” a big code change that nullified previous experience. plan b would’ve been using a (php?) framework.

they tested. workflow; core installation wasn’t going to do it. “everything that we’re relying on is a module” “sounded kinda sketchy, but drupal is just a really small core” “workflow module is better than any workflow [?] I’ve seen.”

content input. “someplace where drupal could use some improvement” using TinyMCE; so many instances can be a drag, their needs are fairly complex, lack of safari support. but it’s enough.

cck. “it’s hard to do much of anything without it.” model the content. “great but weak in how to move content types from one to another” (???)

something he skipped past really fast.

photo handling (image field, image cache, custom slideshow) they’re happy.

taxonomy. when you have so many different kinds of content it’s hard to categorize; move it all together after having different meanings in different CMSs.

external content input. they built their own for (???)

templates. “place where drupal could use the most work” – can do anything, but can’t be done by just any designer, they have somebody really advanced.

multiple sites: all the sites that start with “nw” – not doing it now but in the future.

URL control. “hackable, stable, all those good things” – “surprisingly difficult for cms’s to understand” even in the most expensive/sophisticated. not built into core, using “pathauto” (?)

file type output. simple. multiple databases. not as simple, but migrating has gone with relative ease. (from MT, among other things.) content auditing. good.

overall extensibility. this was the best thing about drupal. “what if we want to do this in the future?” and for anything was someone in the community who was at least thinking about the idea. (I really do wonder if the thing I need, whether anybody’s thought about it.)

benchmark. missed exactly what he said, but saw the “so-so” gesture.

brought in lullabot to answer early questions, to deal with corporate-style unease. “matt’s here to train us, matt wrote the book” also local company fu(z/s?)IQ.

first switched articles. create connections between things. (hm, tags.) everything has a page. (neighborhood, column, author, etc., etc.) (hm, able to cross-reference some of those things that go across categorical boundaries.)

change to different guy. (name?) “not a theming guy” & designer won’t be here. (dang.)

thing loved in drupal over joomla. joomla would’ve got them up earlier, and easier for designers at outset. but at the time the infrastructure was not as strong. drupal required more up front, but with more payoff in the end. (that’s been my impression too.)

used to a really sharp division between admin & content, and the drupal way makes him a little nervous. you can’t just take a default install and throw it out there; can’t close off from the community and updates. (esp. re security)

example of deciding on what templating engine to use. their internal discussion vs. trends in the drupal community. “making sure you’re not digging yourself into a hole that no one will be around to get you out of.”

import content. 4 or 5 taxonomies. lots of use of cck. confusion between what’s a taxonomy and what they’re using cck field for. (hm. seems like a possible pitfall.) still need to do on-the-fly imports from the actual newspaper sites, taking XML. decided it would be easier to export drupal to verity (? search engine) rather than other way around.

deployment process. becomes very difficult with so much in the database; can’t just copy over because of so much user-generated content.

had to do drupal in parallel with some coldfusion stuff. “drupal likes to own everything.” (no kidding.)

first guy talking about “the handbook” (lullabot book?) with some confidence.

integrating login functionality with other sites/services without multiple logins. (SSO, basically)

discovered because of drupal a bunch of crazy server configuration issues. (want to talk to EW about getting a long-term testing server.)

sketch on the board about development server process. Dev -> QA (stable server) -> load-balance production servers. same deal with databases. in the development environment, each developer has their own local install, incl. db. but can also share where that’s appropriate. OTOH, all the “editorial” (content) is done on the live server. which causes its own problems. they don’t have a magic bullet. has heard of something called “autopilot” which isn’t quite there yet. the problem space is really really really complicated. (weird stuff with cck.) working on a project where themer and developer are writing down all the changes as they go to the QA server, and then use the same list when going to production. something about cck UML. (!) (their current process reminds me of when I did a server migration last month. I still missed a bunch of little things, particularly edge cases.) “content copy” module; somebody in the audience got it to work. audience: “custom module as function” (? no idea what she’s talking about.) something with subversion? “views” has an export function. (okay, now I’m lost)

guy from tacoma said that lullabot workshop said they recommend something like that. (way confused, but then again, I’m not sure how much this applies. well, maybe with olyblog. but I don’t think they’d have anything like a development server!) has noticed a lot more chatter about this as drupal gets more “enterprise” attention.

question: xml import stuff? surprisingly simple. (fun with tiny print!) arrays of arrays of arrays, importing into cck(?). (looks tres familiar, maybe from my crazy experience with the college catalog.) use curl to grab some of the content. (I have something I’m doing that with, but not actually dealing with xml as xml, just regexp.)

(missed a bit) something about path redirect, and making all the old URLs permanent. a real pain to do, but worthwhile. (I converted a bunch of pages from underscore to hyphen and/or folders, and yeah, it’s a pain, but I would’ve felt awful if I hadn’t done the .htaccess work.)

can add an include of drupal’s bootstrap file in other php stuff. great for importing.

they have a “nwsource” module that includes all their custom stuff, that includes the xml import; imports to node, etc. “one-click way” “so happy to have made their lives better” (dude, I totally know that feeling.)

missed a bunch of stuff while looking at OlyBlog.

question: anything you needed to strip (esp. from core) for performance? something about sql calls to (? lower?). huge discussion in drupal community. weird thing: can have case-sensitive display username, but actual ids are case-insensitive. easy in MySQL, hard in Postgres. huge, huge problem. lower searches to user table were causing the most problems. (oh, hey, maybe that’s something that would speed up OlyBlog!) when Nanowrimo launched, could only have 200 people logged, w/out those calls could have 1200+ logged in. it’s a very difficult problem to solve. (ah, in further discussion, unless OlyBlog isn’t using MySQL, this doesn’t matter.)

good & bad: not a lot of pragmatism in drupal developer base; leads to holy wars, contentious issues never getting solved, OTOH part of what makes it so flexible.

they’ve submitted a fairly substantial patch to image(?) to include caption & credit.

other guy: big change via module they wrote called “comment tweaks”

they keep all their patches in a separate directory with notes about what issues the patches fix.

questioning: security hardening? a lot at the apache level, deny all to lost of stuff. blocked cron to everything but internal ip. didn’t block admin that way; very difficult to just block admin stuff. even using an admin theme, weird floating back and forth. “the usual” some joking about default ids. keeping up with updates from the site. a pain, but really important.

question from staff: show everybody what’s powered by what module. have abt 30 installed, but that’s not an unusual number. “yeah, blocking our own ads” with flashblocker, inspires a chuckle. cck content type called column, with all the usual types. “author”, “publish date” mean something different to them than to drupal. author of article doesn’t have a drupal identity, is just a content field.

missed a bunch of stuff. “jquery is easier”

neighborhoods, tags, custom tags are all taxonomy. (I think that’s what he said.) cap path? and pathauto, using ideas from SEO consultant. RSS sort-of for free, some weird stuff here. article info feed also includes feeds for author & column. (changes to page related to a specific topic?!)

related articles comes from a node reference.

“printer-friendly” module. “if anybody wants to rewrite a module in the drupal way, please do this one.”

forward module. digg link, but isn’t getting any use. mention from audience of “services” module.

drupal’s comment preview works weird in their experience, changing display name, etc. not sure it was necessary, but get comments “doesn’t look like a drupal site” (which I think is generally a compliment)

crazy stuff with the homepage, which isn’t drupal, except that it kinda is. also crazy stuff with old pages and symlinks and redirect rules. “one of the most painful and interesting parts of the whole process” also “voodoo”

(brain is seriously burning out.)

recommend you don’t launch redone site in pieces!

full multi-site setup, with shared users, taxonomy and roles, but only running one site now. can have a user with different roles on different sites, for example. single database for multisites, with prefixing stuff.

can’t have ampersands in urls in drupal, but they had some crazy stuff with old files. (ah, ampersands. my old nemesis. 😉 )

goofy story with old code and id issues (tid & id; you know, that sounds like some code I’ve found floating around).

web servers are abt 3 yrs old, “dual core something”, bumped up the memory, like mentioned before, had to standardize configuration, particularly to save memory. (hm.) dropped max_requests_per_child a lot. (interesting discussion of threads that’s just a hair over my head.) only turned caching on, not “agressive caching”.

apparently in drupal 6 there’s going to be a config setting to only load modules if the page needs them.

(more talk just over my head about dev vs. production servers)

more simple tuning: move drupal’s htaccess into apache conf so apache doesn’t have to look for htaccess ever. (actually, that’s a pretty good idea.)

“the application of death” (?!) to check whether coldfusion was still running, but in doing that would eat up a bunch of memory, and machines would shut down a couple times a day (? or every couple of days?) without anybody knowing. (!!!!!!) plus extra craziness with a cron rebooting coldfusion. (bad code always makes people so animated!) “omg that code is so evil you have to kill it”

(neat things to think about re what we would need to tune apache for.)

idea that they didn’t do – 2nd server to handle only static files, with something like lighthttpd instead of apache. they didn’t have time, and then didn’t have to, but think it’s a good idea. (plus he mentions the thing from shelley’s graphic books about browsers only handling 4 simultaneous requests. I have to admit again, that C was right in our old argument about that.)

because drupal handles 404s, and that took more work, they had to look at the most common to save processing.

took an approach of “it’s not drupal, it’s us” to start with when troubleshooting & memory use. no global answers to optimization.

missed some more stuff. sounds like maybe as much of a year evaluating & preparing.

question: issues with company and open-source? wasn’t as difficult as he imagined. “if they can see a peer doing it, then that helps” in their case, including the Onion. 🙂 issues with wanting to use a module that’s not done yet, and how to be able to help that happen. “working with” vs. “working for” still a challenge. “I don’t have time to write something as robust as the community can write.”

ack. 8:05, and still going. should I pack it up or stay?!

being a responsible member of the community: writing to drupal standard, returning code to the community, not over-hacking the core, etc. talking to somebody earlier today who didn’t understand why they’d be doing this (the meeting, sharing with people outside the co.)

diligence instilling trust. testing many systems, user stories, bringing in a professional company, etc.

okay, I gotta take off.