Comments on: Data Preprocessing – Lessons Learned

By: Raza

Raza — Sun, 25 Jul 2010 20:28:25 +0000

@OG Dude pre-processing can be pain but it is the MOST important step in any data mining projects. Unless feature selection/extraction and data representation is done right, your DM technique is next to being useless. Spending more time in pre-processing is actually a good thing.

“Worry about the data first before you worry about the algorithm” – Peter Norvig

By: Tweets that mention Data Preprocessing – Lessons Learned « Let x = x -- Topsy.com

Sun, 25 Jul 2010 13:08:48 +0000

[...] This post was mentioned on Twitter by Alyona Medelyan, Tara. Tara said: RT @zelandiya: Data processing – Lessons learned: Tips from #nlproc grad student: http://bit.ly/bpifBY [...]

By: OG Dude

OG Dude — Sun, 20 Jun 2010 10:42:26 +0000

Yeah, I guess you could carve those truisms in stone… pre-processing especially is a b*tch. Frustrating to know you spend more time on squeezing stuff into the right shape than actually doing something noteworthy with the data.

Another thing I find hard to get right is the trade-off between “early optimization” – which is bad and doing a quick and dirty job – which is also bad. I mean most of the times you’ll NEVER use code you wrote for a particular project again but if you’re like me then doing a half-ass job under that premise is like watching people use MS Comic Sans in Powerpoint – it gives you cringes on so many levels…

By: DrNI@AM

DrNI@AM — Wed, 03 Mar 2010 07:18:16 +0000

Congrats for finishing your thesis! And thanks for the advise in this post. It comes too late for me, hence I can confirm most of the points.

We tend to think of pre-processing tools perfect and readily available things… depending on the application, the errors in sentence splitting, tokenization, and tagging multiply. Fixing these things could fill a thesis of its own.