SM01: Web Crawling with JLC parallel execution pattern - experiment reports
Published: 1 November 2017| Version 1 | DOI: 10.17632/5prx8vycr5.1
Contributor:
Goran GrubićDescription
Research project SM01 (Parallel Semantic Crawler for manufacturing multilingual web...) The JLC pattern is running parallel thread for each domain in the Crawl Job domain cue. Objective of the experiment was to find what number of parallel DLC threads (TCmax) results in the highest execution efficiency. The crawlers were run: 2, 4, 6, 8, 10, 12, 14 and 16 parallel DLC threads, against the Sc subset. Other configuration parameters: Load Take per iteration (LT) set to 1 and Page Loads limit (PLmax) set to 30. Sampling period of the resource utilization: 5 seconds.
Files
Institutions
Univerzitet u Beogradu Fakultet organizacionih nauka
Categories
Manufacturing Industry, Serbian Language, Web Site