As a follow up to my open letter on Adobe Certification (which again, never garnered an official response), I have an update to share with my readers. Before I do so though I have a lot of preparatory text I need to get out of the way. The following blog entry is based on a meeting I attended (non-NDA) for Adobe User Group Managers and Community Experts. It was specifically on Certification. This meeting left me frustrated and - frankly - a little bit more angry. I should perhaps have waited before blogging - but I wanted to share the notes while they are still fresh in mind. Just keep in mind that this may not be my most rational response. I'm emotional now (scratch that - I'm angry) and if you want to stop reading, I won't mind.
Note that when I use quotes it should not be considered exact quotes. Rather it is my notes and I wanted to visually differentiate between the slides in the meeting compared to my thoughts.
Software is Adobe's core business, not certifications. Anything they do outside of software though is meant to help the core business.
Very fair point and I agree completely. I wouldn't fault Adobe for a poor grammar choice in their docs for example.
Certifications serve three main purposes: insure adequate technical knowledge for certified instructors
ditto for adobe solution providers
provide credentials to help developers get more business and better jobs
Makes sense to me.
Certification is expensive. Costs include development, validation, localization, publishing, and tracking. The cost is around $50K per exam and every change is a $2.5K cost. (Unless they just turn off a bad question. Adobe does not make money on certifications.
Ok, no big surprise here, but frankly, I don't care. Really, I don't. The whole point of this issue is the quality of the certification and the fact that Adobe is selling something they don't stand behind. When the presenter was pushed on this, he said some internal forces were pushing back against just pulling the exams. Frankly those internal forces are wrong, in my opinion. Writing certifications are hard too. But darnit - if you can't stand behind it - don't sell it.
The creation process involves writing objectives (which is started fresh for each product version). Then they find individuals (like me) both internal and external to help write them. They are then reviewed (again, by folks like me, both internal and external). Based on feedback, Adobe sets a passing score for each exam.
For my thoughts on the write/review process, see the old blog entry.
After publication, how does Adobe verify the certification? They check pass rate. They aim for 60% of folks passing. They check the amount of time it takes to pass the test. They also do a 'point by serial correlation' check where they can look at a question that folks missed even though they passed. That's a good indicator of a bad test. Unfortunately it costs for Adobe to look at these.
Nothing much to say here - but again - I don't care about the costs (admittedly, easy for me to say).
Now we come across the slide that made me see red. Just ask the guys on my IM buddy list. They can't print here what I was saying:
Bad things can happen, and did. The first bullet point of blame was on the writers and reviewers. The last bullet point was on the vendor. The vendor was outsourced and looked good on paper.
So. Um. Can you see where I got angry? Hint - it's the second sentence. To say I was a bit upset would be a major understatement. Myself, and the the two reviewers, were very vocal. I also emailed Adobe. One of my contacts on the CF team, who I will not name, basically said it was the cert divisions problem, not the CF teams problem. So when the blame was laid right down on me personally, I kinda flipped.
Now the presenter, at the end, came back and acknowledged that that wasn't what he meant to do. He personally apologized to me. (And I'm not going to name him because he is an old timer, an Allaire guy, someone I know, and he was in a very bad position here and I don't want to shoot the messenger.) And let's be frank - you guys have seen my code. You know I make mistakes. Hell, I love to blog my mistakes because I figure people can learn from them. (Just ask me about how I messed up this CF server today. It was a real bonehead mistake.)
That being said - that was one of the worst slides in history. The number one bullet needed to be that the blame was squarely Adobe's fault. Stop passing the buck!
When pushed on the "if you won't stand behind it, don't sell it" point, again, the buck was passed, this time to internal forces. Apparently these internal forces won't let them stop selling the certs.
Adobe repaired as much as possible, but did not rewrite the certifications. One test was pulled. One test was not required for instructors anymore (Dreamweaver). One was totally rewrote (InDesign). On another test, it had a lot of issues but Adobe left it be since folks tended to pass because they figured out the right answers.
Not much to say here. The last sentence is a bit hard to believe though. Folks shouldn't have to figure out a certification.
The tests will not be rewritten for the current versions.
Last note - there are no plans (as of yet) for a Spry certification.
I don't know how much I can summarize this. I'm trying to clear my head a bit. I can say it is extremely frustrating. Extremely. I try not to say I have a "big" blog, because I figure as soon as I do my traffic will drop (and frankly, that guy with the big arms is kicking my butt ;) but I did expect to hear something on the blog from my last entry.
It sounds like Adobe really gets that they have a problem. But the #1 thing I got from this meeting was one big giant buck passing. Blame the writers/reviewers (and again, I'm not taking this as personally as I was initially). Blame the vendor. It's the cert team fault. Etc. None of this sounds good to me. I'm more than willing to say I know more about My Pretty Pony than large multinational corporations and economics, but at the end of the day, I don't believe Adobe is doing the right thing here. I've been called an Adobe Fan Boy on more than one occasion but I got to speak up here.
Archived Comments
This sucks - esp. from a company that has been presenting itself as a company that listens to the community. I was in a meeting today at my new company and they'll actually pay for certification - but I'm not going to do it now out of principle!
I'm just guessing here but if you get caught with the buck, you probably get fired, so everyone just passes it in a desperate act of preservation. Just thought I'd start things off here :)
@ Jim not to mention you might have to "figure out the right answer". Now I know in programming exams that is generally the point but we all know what I mean right? It does suck though as I would have liked to do the certification myself but really what's the point if the company that issues them has issue with them?!
Just finished watching the part of the preso in question. I see why your pissed honestly, but I don't think he was being specific to CF (and your participation) but to certs for all the products. Maybe there was a bad review or reviewers for some other line?
It would have been great for him to properly respond to your concerns in the chat and the previous blog post. I am really surprised at that honestly. If there is good feedback sent back but it goes to a black hole, then that is indeed a real problem. Maybe that is something they can work on during the whole openness effort. Good quality format for feedback that goes to the right person and is measurable...
adobe doesn't make money on certifications??? so they don't think by getting someone certified in adobe products that this person is more likely to stick with adobe for the future therefore buying upgrades... new adobe software... etc? creating all his client sites w/ adobe products, therefore getting even MORE sales for adobe?
yikers.
@fuzie: I believe the $$ issue was being very strict on the cert itself. Ie, it costs 50k, and they sell N*whatever which earns them < 50k. All the other stuff is a bit hard to measure.
@joshua: The presenter, at the end, made it VERY clear it wasn't a personal attack on myself. I leave it up here because I honestly feel its part of a bigger issue of not accepting responsibility for the cert.
I do agree though - I do wish Adobe had responded to my post. I do NOT want to sound like a diva here. I get a LOT from Adobe. Early sneak peaks. Software giveaways, etc. But again I've been surprised by the lack of response on the blog.
Well, I think they should spend what it takes to create a worthwhile cert program... give someone else the rights to do so... or don't have one at all.
A smart business leader would see that though there may not be direct returns on investment, there are infinite benefits to the company overall.
I would bet that what it would cost for them to certify ME comes out to far less than what I would then continue to spend on dev tools - software that I might opt to get elsewhere in the future if I wasn't already invested in Adobe in kind.
What is really sad here is that Adobe is not being asked to invent the wheel here. ORACLE and Microsoft have already been doing the certification thing, much better than Adobe has, for a very long time.
Education in IT is important - period - and Adobe just doesn't get it. I tried to hire an entry to mid level CF developer at our company (based in Indianapolis) for 2 months and guess what - they don't exist. Why? Because there is no new blood when it comes to CF. Why? Because Java, PHP and .NET have a long tradition of getting into universities via giving shit away or just because they're free offerings to begin with. Adobe does not care about education or certification because they're still selling product (for now). ColdFusion may already be doomed because it is no longer a dominant player in the app server market and it is not taught anywhere, grads get jobs then they rise through the corporate ranks and never they never consider CF as a solution at their respective companies. How did you learn CF? Self taught or taught by others at work? Maybe you learned the hard way like I did - by building apps in it and looking through the CFWACK books to make them better. Maybe you were lucky enough to find this blog or some others right off the bat.
At any rate, Adobe doesn't care about certifications or finding new users. When was the last time you saw a CF add in Information Week? I spent an hour on their website trying to map out CF and FLEX certification details and came to the conclusion that they don't care. If they did care then I would argue that the site wouldn't have been such a mess. Go to Microsoft or ORACLE if you want to see a software company that understands that a developer's career is often tied to a software vendor and how much they can prove they know about that software vendor's products and overall technology platform. Recently I've developed serious worries about building a career around CF. I'm taking my personal certification/training budget and using it on C#.
CF is my favorite and it is such a productive environment but it may be the next Novell. Do any of you want to cross your fingers and stake your careers on a niche? Is Adobe really spending any money to recruit new developers or show the world their great product? No, their completely insulated and surrounded by folks who already love their stuff. They're not out there trying to get people to switch from other solutions to CF. Show me one shread of evidence that they are and I'll shut up and post a retraction to that statement.
When you say Adobe do your colleagues and clients immediately think Photoshop and Acrobat? Do they even know what CF is? Who's fault is that?
Well, I want to applaud you for speaking out about this! Your previous "open letter on Adobe Certification" post has affected me personally. I had been planning on getting certified, but had never gotten around to it, and that post inspired me to kick it in gear and get the CF 7 cert while I could. I found 1 testing center (out of 5) in my area that still offered the CF 7 exam through June 30th, and I was able to get in there and take the test on the last day! If it wasn't for you, I would have waited around and been stuck with CF 8 or nothing!
Thanks Ray!
Since the preso was not under NDA, you could probably post a link to it for those who couldn't attend...
You would think that if they are footing a $50k bill for this thing that they would at least want something quality in return. They should search the contract with the outsourcing company for some sort of "if you're not satisfied" clause. Of course it's probably too late as it is a done deal.
@Andy - Adobe did recently announce some educational related news in regards to CF, and there is the whole open-source CFML initiative... the question is - is it too little too late?
This will be a good test - if Adobe really does listen to us they will fix this.
@Jim
I'm well aware of this and in my opinion it is about 5 years too late. The blame can't be completely put on Adobe as Allaire and Macromedia had the same shot and blew it.
You might say that the culture and image of our product of choice has evolved into one of being a maverick. If you think that's pretty cool just ask Borland how that's going for them.
Anyway I'm a sore loser and I'm pissed off that my product of choice is losing. Losing market share, losing mind share and eventually it will start losing money and it will get dropped. And all of the truly smart people like yourself and Ray and everyone else will go off into the sunset and master something else.
Lately being a CF proponent seems probably a lot like being a Detroit Lions fan. The team keeps getting worse and the front office keeps f@@king it up even worse. I wouldn't know about that because I'm a Colts fan and we finally got our act together.
What's sad is that this is a terrific, feature rich product with an inventive group of users (dwindling in numbers) who would lay down in traffic for it. CF is way better than the competition in many ways. But the folks who have made it great over the years aren't that good at the business side of this. How in the hell do you lose the thrown when you're the first one there? (CF existed before ASP and PHP)
@Andy,
I would disagree with you on this. I think the folks inside Adobe (specifically the CF team) are "getting" it more and more and come with more business savvy at the same time. People like Mike Nimer, Tom Jordahl (and other older CF engineers) have given way to newer developers and with Jason Delmore and Adam Lehman (controversy aside) on the product side, I think we can see more and more out of the CF team.
@Sami,
If they're "getting it" and CF8 was released a year ago then why no marketing campaign that reaches for new users nor any mainstream advertising? Why did it take just now before it dawned on anyone that CF is never front and center at universities or training centers (the place where many developers are born) and that maybe if it were given to colleges they might use it and expose their students to it. Oh but wait, how could said colleges teach classes on it if there aren't any decent training materials or certification tracks that mean anything?
I live in the 12th largest city in the US and ran an ad for an entry to mid level CF developer for 4 months and had 2 qualified candidates - each of whom had more than 8 years of experience - not exactly "mid level" or new blood.
So what is it that we'll "see more and more of"? More people picking up .NET or PHP? It took me many many months to get this upset. Do you think that I want to abandon a world class product like CF in favor of other products that are not as productive? No way, but if I want to continue to grow my business I'll have to because I can't seem to hire anyone and Adobe doesn't seem to place too much emphasis on growing the user base, marketing, education, training or certification.
@Andy,
This isn't the place to get into this here, but I understand where you are coming from. I was teaching CF in 2001, so I understand where the education aspect lacks. I've been in talks with my university to bring RAD web tools like CF and RIA into the curriculum. Also, as a hiring manager myself, I've seen the lack of talent out there as well. I've interviewed hundreds over the years. Lots of horror stories.
At the same time, I've noticed some changes inside Adobe. I've been a beta tester since version 6, and I've never had more access to make a difference myself. So maybe what I meant by "see more and more of" is what I'm seeing on the inside, and hopefully most others will see on the outside.
Adobe is a very mature company, and at the same it doesn't react quickly sometimes. In the post-Google age, its becoming less of an option to think old school.
So what am I saying? Don't keep frustrated, have some patience: keep the faith. With Railo, and changes announced for Centaur, CF is going to be accessible and "complete" than ever before; someone is listening... finally!
I always thought getting certified was a good thing. I have my CFMX cert and had planned to do the upgrade to CF8 and even encouraged my boss to encourage the development team to get certified.
Why? Well if nothing else it "could" give the client a little bit of a warm and fuzzy knowing the entire team is certified, but that assumes that the certification is actually worth something that its not easy to get.
I will probably end up just getting it anyway but these revelations bother me. Certifications should be hard to get and worth the effort in getting.
As far as I am concerned, if Adobe are acknowledging that it is broken - the statement "The ColdFusion MX Developer Certification represents a professional level of expertise and demonstrates the competency of a ColdFusion MX developer." is a lie.
Developers generally* have a strong sense of honour, and a highly analytical mind. We are engineers; if something is broken, it must be fixed or torn down. I find that developers find middle-ground a painful place to be.
Adobe through certifications have poured away credibility. It takes a hundred bits of good news** to out-weigh one bit of bad.
Adobe - remove the certifications, rewrite them, and then re-release them.
Ray - I read your article - the anger was creditable.
Joel
* Apologies for generalising
** e.g. the bbc statement yesterday
Out of curiousity, how many of you have taken the BrainBench coldfusion exam?
A headhunter had me take it a few weeks ago... most of the questions were pretty obscure, but I did a lot better than I'd expected:
ColdFusion MX
Score: 4.69
Percentile: Scored higher than 99% of previous examinees
Account Percentile: Scored higher than 99% of 658 examinees within this account
Proficiency Level: Expert (Master)
Haven't taken the CF one, but it sounds like it's not worth bothering with. Is there another one out there that is? Most people I mentioned Brainbench to said they'd never heard of it....
For shame, Adobe.
I've been dreading this exam, but unfortunately, it's a necessary evil for me to advance in my job.
Thanks for being vocal about this Ray.
ok - so someone with clout start a community drive CF knownledge meter - I don't have the details thought out - just brainstorming.
A process could be something like design/create an application to do foo. Requirements: must use transfer/MG(or whatever)
4/5 panel person reviews the submissions then awards points for each person.
So you pick up points by participating in challenges or submitting work or by prior experience points from graders.
Once you build up enought points, we get a shiny plastic badge and a piece of paper from the community that says you know or possess skils to develop apps the CFML.
again - just a thought I had. If it is close to one your ideas, I'm a bad reader so I overlooked yours. I apologize in advance to that.
@Andy S:
I consider myself jr going on midlevel myself, and I can see what you're saying; I feel kinda unique, in that the majority of CF developers I know have been at it for 5 years minimum.
I've been at it fulltime for about 3. (and with that comes learning CF by myself, learning bad habits, breaking bad habits, trial, error, etc)
I too hope that somehow Adobe really makes a push for CF very soon. While I may still be so early in the game I probably still qualify as a 'fanboy', I'd rather not have to move to .Net.
Please, Adobe? Do something?
@JC - Nice to meet another Brainbench-er. Our scores are VERY similar.
I had a headhunter ask me to take a test yesterday - and was I SADLY disappointed. First - their customer is running CF7 and migrating to 8 - let's start there.
So the "test" to "prove" my resume - was based on CF SERVER 4.5 and CF STUDIO 4.5. Not only were the questions SERIOUSLY dated - but some of it was out-and-out BROKEN. (ex. missing "=" signs between tag attributes and their values).
If this had been some hand written test - I might have been less shocked. But this was from a "testing company" that provides "certifications". NOT IMPRESSED to say the least.
Well, I can't argue or take exception to anything that Ray states. I, too, long for the days of the ColdFusion certification program being under the wings of Allaire or Macromedia. Adobe has been a major obstacle instead of what should be viewed as a win-win relationship.
However, if you do want to or have to (job promotion, training requirement, etc...) take the CF8 certification exam, you might want to consider getting CF8 Exam Buster which I just released (http://centrasoft.com). I won't plug any further. Thanks, Brian
am adobe certified but nowadays lot of people cheat during exams so please do not give much credentials.. it is like 3 people writing exam and 6 people googling out the answers for them..it happens in most software company's in house exam facilities
basically do not believe in certifications if they wrote it outside pearsonvue or similar acredited centers.. place zero value to software company in house test facilities that nowadays adobe allows.. i personally lost so much money taking my certifications but seeing people cheating at in house facilites, i feel betrayed... omg