If it's your own website, you don't need a crawler. You have the full content in a nicely structured set of folders or - more likely - in a well-structured database. Crawlers rely on interpreting html like a browser does. No need for that in that case.
Just use tools for file searching and database querying.

Or am I missing something?