This is an old revision of the document!
Tools
This is an incomplete list of tools you can use to build corpora from the web.
Complete pipelines
- BootCaT – bootstrap specialized corpora and terms from the web
De-duplication
- Shared ngram collector – Perl script useful for near-duplicate detection
- Onion – a tool for removing duplicate parts from large collections of texts.
Boilerplate removal
- jusText – a tool for removing boilerplate content
- WebContentExtractor – a tool for extracting content from web pages
- the PotaModule (a Perl module that is intended to perform “boilerplate” stripping and other forms of HTML document filtering and extraction) is available in the BootCaT toolkit (see link above).