Thank you for being a valued part of the CNET community. As of December 1, 2020, the forums are in read-only format. In early 2021, CNET Forums will no longer be available. We are grateful for the participation and advice you have provided to one another over the years.

Thanks,

CNET Support

Question

A cross-platform content grabber

Oct 21, 2015 4:02AM PDT

Alright, I know that this might not be the right term here, but I meant the mechanism sites like indeed.com utilize to grab contents off of sites all across the world wide web and then automatically post onto their own sites. What's this tool/mechanism? I don't think a web crawler is able to perform the latter task, but I could be wrong though

Discussion is locked

- Collapse -
Answer
Re:grab
Oct 21, 2015 5:12AM PDT

What parts of the whole process don't you understand?
1. Grabbing content.
2. Publishing content.

Kind of automatic copy/paste. If you can do it in your browser, a program can do it also.

Kees

- Collapse -
what and what?
Oct 21, 2015 8:29PM PDT

the whole process of what?
And you sound like as if you have explained whichever idea you were trying to explain repeatedly judging by your tone? which is not the situation, m'dearWink

- Collapse -
Re: process
Oct 22, 2015 12:12PM PDT

I meant the whole process "to grab contents off of sites all across the world wide web and then automatically post onto their own sites."

In the first 3 lines i didn't try to explain anything. That was just a question or two.
In the last line I explained in layman terms what they do: "kind of automatic copy/paste: copy from a site across the web and pasting onto their own site." And be sure that's the first time I explained it like that.

Kees

- Collapse -
so what are you trying to say?
Oct 24, 2015 4:03AM PDT

Why do you question me if I understand my OWN question? Or are you trying to say that this is what a typical web CRAWLER does, from what i know if it is not.

- Collapse -
Re: web crawler
Oct 26, 2015 1:04AM PDT

A typical web crawler from google copies contents from a site into googles database. It's just as easy or even easier to copy it to a site than to copy to a database.
So, yes, its a variant of a web crawler.

Kees

Post was last edited on October 26, 2015 11:23 AM PDT