Livebook

Blue system‘s Specs:
CPU: Intel(R) Core(TM) i7-2600K CPU @ 3.40GHz, 4 cores, 8 threads, 32 nm, 1 mb L2 cache, 8 mb L3 cache
GPU-1: Intel(R) HD Graphics Family, Onboard
GPU-2: Inno3D Geforce GTX 570, 1.28 GB GDDR5 3800MHz, 732MHz Graphics Clock, 480 CUDA cores, 320 Bit
Ram: Corsair, 8Gb DDR3, 9-9-9-24, 1600 MHz
M.b: Asus, P8Z68-V
HDD-1: SSD, OCZ-VERTEX2, 128 Gb
HDD-2: Western Digital Caviar Green, 1Tb, 64 MB Cache, SATA 6.0Gb/s
OS: Windows 7 Ultimate, 64-bit
Windows Experience Index: 7.6

Red System’s Specs:
CPU: Intel(R) Core(TM) i7-2700K CPU @ 3.50GHz, 4 cores, 8 threads, 32 nm, 1 mb L2 cache, 8 mb L3 cache
GPU: Inno3D Geforce GTX 550 Ti, 2 Gb GDDR5 4100Mhz, 920MHz Graphics Clock, 192 CUDA cores, 192 Bit
Ram: Apacer, 8Gb DDR3, , 9-9-9-24, 1600 MHz
M.b: MSI, P67A-GD55
HDD: Seagate Barracuda, 1TB, 64MB Cache, SATA 6.0Gb/s
OS: Windows 7 Ultimate, 64-bit, Service Pack 1
Windows Experience Index: 5.9

I benchmarked them using PCMark Vantage Professional 64-bit Edition. It took almost 1 hour and 30 minutes and here’s the result:

The Blue System was the front runner and had the ace up it’s sleeve. What was the ace ?!!! Perhaps the SSD drive ! Although the Red System’s CPU was more powerful, but it fought a loosing battle. The Blue System’s HDD performance was amazing !!!

You can find the detailed benchmark result in excel format here:
Blue System’s Benchmark Result – Excel
Red System’s Benchmark Result – Excel

I’ll post more comparisons in the coming days. The next one should be a Nvidia/ATI test using the Blue System.

Kind of a late submission and maybe a late time for user votes, but I like what I did. Writing 30 procedures in over 800 lines of code and comment. That made me quite busy this week. Coding, searching, learning, failing, debugging, documenting, testing… Let’s say I’m quite pythonised !!!

I called it feature pack, because it’s a pack of functions that adds lots of functionalities to your existing search engine and/or crawler.

Here I introduce some of them briefly:

• Virus and Malware checking procedure: Checks urls against Virus and Malware, based on the open source ClamAV engine.
• Phishtank Antiphishing: Check sites against Phishtank antiphishing database.
• Google safe browsing Api: Allows you to check URLs against Google’s constantly updated lists of suspected phishing and malware pages.
• Spell suggestion: checks words and sentences spelling and outputs the correct form, based on the Google dictionary.
• Related Searches: Shows related searches of a given keyword based on the Yahoo database.
• robots.txt finder: find the location of the robot.txt file based on the given url.
• robots.txt parser: BE POLITE !!! Parse the robots.txt file and returns a dictionary of all the allowed and disallowed urls for each useragent, along with sitemap and crawl delay.
• Robots <META> tag: Parse the HTML <META> tag of a given url that tells robots not to index the content of a page, and/or not scan it for links to follow.
• Can Crawl: This procedure takes your crawler user-agent and a url as an input and tells you whether you can crawl the page or not, based on the information of the robots.txt file and meta robots tag.
• Sitemap Parser: Parse an xml sitemap and return a dictionary of all the urls that can be crawled and the change frequency, priority and last modification date of each url.
• Save Bucket: Saves a bucket into given path and compresses it using gzip.
• Load Bucket: Loads a gzipped bucket from a given path and returns a dictionary.
• Url Truncate: Truncates a url to a character length. You can specify the length with the limit argument.
• Title generator: Takes a url as input, fetches the page, matches the <title>…</title> tag and returns it.
• Description generator: Takes a url as input and generate the pages description, based on the description meta tag or body content.
• Alexa Rank: Returns the Alexa ranking of a given url. for more information about Alexa rankings, visit alexa.com. You can use alexa rank for computing webpages ranks.
• Better Get All Links: Return the dictionary of all the links that are in the passed string, with link text and rel attribute.

Also you can find other useful procedures in the code, like:

• get_page() for fetching urls,
• check_response() for Checking server response code,
• download() for downloading and storing files,
• post_data() for passing HTTP Post requests to webpages and retriving them,
• json_decode() and json_encode for decoding and encoding JSON,
• gz_read() and gz_write() for working with gzipped compressed files,
• base_url() and raw_url() for modifying urls,
• strip_tags() for stripping HTML tags,
• remove_extra_spaces() for removing more than one consecutive white space,
• truncate() for truncating a string to a character length.

This code is published under Creative Commons Public License. You can download the search engine feature pack here.

Feel free to remix, tweak, and build upon this code, as long as you credit it’s license and license your new creations under the identical terms.

Also you can find it on pastebin.

Don’t forget to check my existing search engine Paleez.