Question

Hi, I have the problem statement for a project and would like to know what type of programming knowledge is required to do this. Here it is: "A program that will collect data from a website user by user. Suppose there is a school web page where you can insert you number and a new page will come up and show your profile info. Given a certain range of roll numbers (ie. 1025-1125) the program will automatically go through 1000 student ids and save their info pages in a file. It should also incorporate a search engine that will scan a given info page for a specific info (ie. find all people with t

Answer 1

Question continued: (ie. find all the people with the word "fulltime student" written in their page and thus finding all the fulltime student. Also may be to find students of the same last name etc.) and will be able to take specific action according to that." --This is the basic idea of the program. I have basic C knowledge and have idea about how make search functions, but how do I handle the network, webpages and file saving/data storing parts? What type of programming knowledge is required for this?? Thnx

Answer 2

It's not actually so difficult, you need to do a basic web site obviously, standard web stuff. You need a database you can do some pretty simple queries on and something to script it all with.

Personally, I would use App Engine with Python, but that's just me.

You would need to know enough HTML and CSS to build the pages. This is a cakewalk; you could learn it as you made the pages with it, it's so straight forwards. Just use tables. It wouldn't be the most elegant source code, but it'd work ok. Then you could use Python to write the rest. You could do the same with Java if you like.

You app would only need a few query definitions in a few request handlers. You'd have to learn some Python or Java and learn the basics of how to build on App Engine, but that's about the most direct route I can think of for building what you described and getting it online properly.

Are you actually look to build this app or are you looking at it in theory for academic reasons?

Answer 3

Nah Carl, he's looking to build a spider. It's not too hard actually, what you need to do is learn how to use sockets and use it to connect to the website and read information. Once you can get at the information, store everything in a database. Then do your searches on the database, to do so of course you will have to know html and perhaps css, unless the website is in flash, which will make it a lot harder to pull information.

Shouldn't be too hard to do, but it will take some time. Remember that what your building is a spider and you can search for spiders that already exist to get hints on how to build your own.

Answer 4

@lando
Nice work, I completely misread his question.